[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:11 dist-test-slave-pww0 krb5kdc[30859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:11 dist-test-slave-pww0 krb5kdc[30859](info): set up 2 sockets
Aug 10 23:38:11 dist-test-slave-pww0 krb5kdc[30859](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 23:38:12.645061 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.495s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:12 dist-test-slave-pww0 krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:12.860070 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:33915
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:13.215667 30881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:13.216025 30881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:13.216167 30881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:13.223992 30881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:13.224146 30881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:13.224233 30881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:13.224315 30881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:13.224395 30881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:13.235008 30881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:33915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:13.236523 30881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:13.238025 30881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:13.251127 30888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:13.251097 30901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:13.251065 30890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:13.252012 30881 server_base.cc:1034] running on GCE node
I20260810 23:38:13.252935 30881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:13.254653 30881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:13.255911 30881 hybrid_clock.cc:648] HybridClock initialized: now 1786405093255849 us; error 120 us; skew 500 ppm
I20260810 23:38:13.256345 30881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:13 dist-test-slave-pww0 krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405093, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:13.264532 30881 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:13.266474 30881 webserver.cc:460] Webserver started at http://127.30.33.62:38229/ using document root <none> and password file <none>
I20260810 23:38:13.267304 30881 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:13.267445 30881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:13.267823 30881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:13.271379 30881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "7b13b537bf654a4c8b8e7460cf1f35ed"
format_stamp: "Formatted at 2026-08-10 23:38:13 on dist-test-slave-pww0"
server_key: "7245146f06c05e4c69ea67058eececc4"
server_key_iv: "11dcbf117cf335c726b759cf0b8f241a"
server_key_version: "encryptionkey@0"
I20260810 23:38:13.272390 30881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "7b13b537bf654a4c8b8e7460cf1f35ed"
format_stamp: "Formatted at 2026-08-10 23:38:13 on dist-test-slave-pww0"
server_key: "7245146f06c05e4c69ea67058eececc4"
server_key_iv: "11dcbf117cf335c726b759cf0b8f241a"
server_key_version: "encryptionkey@0"
I20260810 23:38:13.278692 30881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 23:38:13.282907 30919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:13.284487 30881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:38:13.284689 30881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "7b13b537bf654a4c8b8e7460cf1f35ed"
format_stamp: "Formatted at 2026-08-10 23:38:13 on dist-test-slave-pww0"
server_key: "7245146f06c05e4c69ea67058eececc4"
server_key_iv: "11dcbf117cf335c726b759cf0b8f241a"
server_key_version: "encryptionkey@0"
I20260810 23:38:13.284900 30881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:13.304180 30881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:13.309177 30881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:13.309500 30881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:13.329485 30881 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:33915
I20260810 23:38:13.329542 31017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:33915 every 8 connection(s)
I20260810 23:38:13.331007 30881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:13.332377 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 30881
I20260810 23:38:13.332720 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:13.334686 30852 external_mini_cluster.cc:1428] Setting key 586f3e452cea746643c04d2fa4c6c6ee
I20260810 23:38:13.335340 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:13.347878 31021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed: Bootstrap starting.
I20260810 23:38:13.351554 31021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:13.352845 31021 log.cc:824] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:13.356582 31021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed: No bootstrap required, opened a new log
I20260810 23:38:13.364888 31021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33915 } }
I20260810 23:38:13.365372 31021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:13.365477 31021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b13b537bf654a4c8b8e7460cf1f35ed, State: Initialized, Role: FOLLOWER
Aug 10 23:38:13 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:13.366313 31021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33915 } }
I20260810 23:38:13.366690 31021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:13.366858 31021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:13.367066 31021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:13.370115 31021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33915 } }
I20260810 23:38:13.370802 31021 leader_election.cc:302] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b13b537bf654a4c8b8e7460cf1f35ed; no voters:
I20260810 23:38:13.371544 31021 leader_election.cc:288] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:13.371740 31033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:13.372483 31033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [term 1 LEADER]: Becoming Leader. State: Replica: 7b13b537bf654a4c8b8e7460cf1f35ed, State: Running, Role: LEADER
I20260810 23:38:13.373204 31033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33915 } }
I20260810 23:38:13.375097 31021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:13.376644 31038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33915 } } }
I20260810 23:38:13.377066 31038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:13.382385 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:13.339088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.30.33.62:33915)
0810 23:38:13.340005 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:13.340023 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:13.341303 (+ 1280us) server_negotiation.cc:407] Connection header received
0810 23:38:13.350589 (+ 9286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:13.350617 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:13.351106 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:13.351588 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:13.355829 (+ 4241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:13.357288 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:13.359362 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:13.360208 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:13.360431 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:13.367792 (+ 7361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:13.367807 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:13.367831 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:13.367908 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:13.372238 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:13.373709 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:13.373720 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:13.373729 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:13.374011 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:13.374727 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:13.374733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:13.374736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:13.376009 (+ 1273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:13.379659 (+ 3650us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:13.380065 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 23:38:13.380653 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":415,"threads_started":1}
I20260810 23:38:13.382418 31036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b13b537bf654a4c8b8e7460cf1f35ed. Latest consensus state: current_term: 1 leader_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b13b537bf654a4c8b8e7460cf1f35ed" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33915 } } }
I20260810 23:38:13.383015 31036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:13.386751 31042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:13.392794 31042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:13.403570 31042 catalog_manager.cc:1347] Generated new cluster ID: 9aecb258908c4729be708a663030dec7
I20260810 23:38:13.403710 31042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:13.423542 31042 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:13.425081 31042 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:13.443483 31042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b13b537bf654a4c8b8e7460cf1f35ed: Generated new TSK 0
I20260810 23:38:13.444644 31042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:13.508227 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33915
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:13.861801 31066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:13.862133 31066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:13.862268 31066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:13.869837 31066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:13.869983 31066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:13.870142 31066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:13.880461 31066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:13.881829 31066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:13.883296 31066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:13.897136 31079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:13.897172 31078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:13.897460 31082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:13.897953 31066 server_base.cc:1034] running on GCE node
I20260810 23:38:13.898589 31066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:13.899627 31066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:13.900840 31066 hybrid_clock.cc:648] HybridClock initialized: now 1786405093900786 us; error 46 us; skew 500 ppm
I20260810 23:38:13.901177 31066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:13 dist-test-slave-pww0 krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405093, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:13.904498 31066 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:13.906037 31066 webserver.cc:460] Webserver started at http://127.30.33.1:46633/ using document root <none> and password file <none>
I20260810 23:38:13.906765 31066 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:13.906893 31066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:13.907207 31066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:13.910753 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "59d78c17b92b4264ba3dff4d014c353a"
format_stamp: "Formatted at 2026-08-10 23:38:13 on dist-test-slave-pww0"
server_key: "d8f4ba174cf682b5dc41b58dd470a36c"
server_key_iv: "bb1943b7d13178fe658d4245926dd485"
server_key_version: "encryptionkey@0"
I20260810 23:38:13.911832 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "59d78c17b92b4264ba3dff4d014c353a"
format_stamp: "Formatted at 2026-08-10 23:38:13 on dist-test-slave-pww0"
server_key: "d8f4ba174cf682b5dc41b58dd470a36c"
server_key_iv: "bb1943b7d13178fe658d4245926dd485"
server_key_version: "encryptionkey@0"
I20260810 23:38:13.917310 31066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:38:13.921080 31094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:13.922497 31066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:38:13.922670 31066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "59d78c17b92b4264ba3dff4d014c353a"
format_stamp: "Formatted at 2026-08-10 23:38:13 on dist-test-slave-pww0"
server_key: "d8f4ba174cf682b5dc41b58dd470a36c"
server_key_iv: "bb1943b7d13178fe658d4245926dd485"
server_key_version: "encryptionkey@0"
I20260810 23:38:13.922884 31066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:13.952883 31066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:13.957746 31066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:13.958046 31066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:13.959174 31066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:13.960825 31066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:13.960906 31066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:13.961022 31066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:13.961087 31066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:13.993309 31066 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:36683
I20260810 23:38:13.993374 31321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:36683 every 8 connection(s)
I20260810 23:38:13.994745 31066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:14.002642 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31066
I20260810 23:38:14.002914 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:14.003669 30852 external_mini_cluster.cc:1428] Setting key f2de903d66dca89ff66b9fa7fe5a8946
Aug 10 23:38:14 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405093, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:14.016870 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:13.997610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:34509 (local address 127.30.33.62:33915)
0810 23:38:13.998070 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:13.998074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:13.999173 (+ 1099us) server_negotiation.cc:407] Connection header received
0810 23:38:14.001133 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:14.001138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:14.001223 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:14.001335 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:14.003312 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.004230 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.006254 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.006691 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.006801 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:14.011022 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:14.011028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:14.011031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:14.011065 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:14.013609 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:14.014396 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:14.014400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:14.014402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:14.014534 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:14.014997 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:14.015001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:14.015003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:14.015297 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:14.015441 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:14.016245 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 23:38:14.016494 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260810 23:38:14.017958 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:13.998242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33915 (local address 127.30.33.1:34509)
0810 23:38:13.998989 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:13.999031 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:14.000822 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:14.001549 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:14.001558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:14.002120 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:14.003055 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:14.003089 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.004430 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.004434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:14.006081 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:14.006098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.006941 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.006948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:14.007256 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:14.008064 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:14.008094 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:14.010772 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:14.013765 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:14.013775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:14.013792 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:14.014265 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:14.014657 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:14.014661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:14.014664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:14.014876 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:14.015430 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:14.015439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:14.015844 (+ 405us) client_negotiation.cc:769] Sending connection context
0810 23:38:14.016121 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 23:38:14.016721 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":293,"threads_started":1}
I20260810 23:38:14.019743 31322 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33915
I20260810 23:38:14.020128 31322 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:14.021129 31322 heartbeater.cc:511] Master 127.30.33.62:33915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
I20260810 23:38:14.023629 30953 ts_manager.cc:194] Registered new tserver with Master: 59d78c17b92b4264ba3dff4d014c353a (127.30.33.1:36683)
I20260810 23:38:14.025215 30953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:34509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:14.051090 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33915
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:14.405335 31340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:14.405646 31340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:14.405781 31340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:14.413419 31340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:14.413560 31340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:14.413722 31340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:14.424216 31340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:14.425616 31340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:14.427073 31340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:14.440955 31355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:14.441102 31347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:14.441056 31351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:14.442040 31340 server_base.cc:1034] running on GCE node
I20260810 23:38:14.442763 31340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:14.443987 31340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:14.445235 31340 hybrid_clock.cc:648] HybridClock initialized: now 1786405094445164 us; error 60 us; skew 500 ppm
I20260810 23:38:14.445580 31340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:14 dist-test-slave-pww0 krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405094, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:14.448966 31340 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:14.450637 31340 webserver.cc:460] Webserver started at http://127.30.33.2:34577/ using document root <none> and password file <none>
I20260810 23:38:14.451385 31340 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:14.451546 31340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:14.451922 31340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:14.455034 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "1ee0f3a1e450474ca371888cb249267f"
format_stamp: "Formatted at 2026-08-10 23:38:14 on dist-test-slave-pww0"
server_key: "4e05ed459d58e230cead33ca4eacedc4"
server_key_iv: "8c3042ab0a93b2cc173fa9141f145c10"
server_key_version: "encryptionkey@0"
I20260810 23:38:14.455881 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "1ee0f3a1e450474ca371888cb249267f"
format_stamp: "Formatted at 2026-08-10 23:38:14 on dist-test-slave-pww0"
server_key: "4e05ed459d58e230cead33ca4eacedc4"
server_key_iv: "8c3042ab0a93b2cc173fa9141f145c10"
server_key_version: "encryptionkey@0"
I20260810 23:38:14.461688 31340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:38:14.465850 31367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:14.467581 31340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:14.467798 31340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "1ee0f3a1e450474ca371888cb249267f"
format_stamp: "Formatted at 2026-08-10 23:38:14 on dist-test-slave-pww0"
server_key: "4e05ed459d58e230cead33ca4eacedc4"
server_key_iv: "8c3042ab0a93b2cc173fa9141f145c10"
server_key_version: "encryptionkey@0"
I20260810 23:38:14.468031 31340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:14.489827 31340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:14.494665 31340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:14.494999 31340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:14.496181 31340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:14.497818 31340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:14.497910 31340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:14.498049 31340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:14.498104 31340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:14.536643 31340 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:38493
I20260810 23:38:14.536722 31570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:38493 every 8 connection(s)
I20260810 23:38:14.538008 31340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:14.543812 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31340
I20260810 23:38:14.544006 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:14.544445 30852 external_mini_cluster.cc:1428] Setting key 642fc76fb772c81ae48719e06486c7ee
Aug 10 23:38:14 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405094, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:14.559906 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:14.540966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:59869 (local address 127.30.33.62:33915)
0810 23:38:14.541473 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:14.541477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:14.542530 (+ 1053us) server_negotiation.cc:407] Connection header received
0810 23:38:14.544544 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:14.544550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:14.544645 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:14.544802 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:14.546602 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.547380 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.549080 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.549535 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.549655 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:14.553647 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:14.553653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:14.553656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:14.553703 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:14.556488 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:14.557361 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:14.557367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:14.557371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:14.557538 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:14.558011 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:14.558017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:14.558020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:14.558316 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:14.558459 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:14.559217 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 23:38:14.559492 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260810 23:38:14.561050 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:14.541577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33915 (local address 127.30.33.2:59869)
0810 23:38:14.542347 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:14.542402 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:14.544243 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:14.545004 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:14.545013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:14.545699 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:14.546418 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:14.546442 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.547578 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.547582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:14.548913 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:14.548927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:14.549789 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:14.549796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:14.550032 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:14.550733 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:14.550765 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:14.553356 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:14.556669 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:14.556680 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:14.556697 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:14.557178 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:14.557657 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:14.557662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:14.557665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:14.557887 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:14.558453 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:14.558462 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:14.558838 (+ 376us) client_negotiation.cc:769] Sending connection context
0810 23:38:14.559106 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 23:38:14.559762 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":304,"threads_started":1}
I20260810 23:38:14.562631 31572 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33915
I20260810 23:38:14.563062 31572 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:14.564227 31572 heartbeater.cc:511] Master 127.30.33.62:33915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
I20260810 23:38:14.566226 30953 ts_manager.cc:194] Registered new tserver with Master: 1ee0f3a1e450474ca371888cb249267f (127.30.33.2:38493)
I20260810 23:38:14.567118 30953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:59869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:14.592934 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33915
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:38:15.028733 31322 heartbeater.cc:503] Master 127.30.33.62:33915 was elected leader, sending a full tablet report...
W20260810 23:38:15.074879 31582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:15.075192 31582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:15.075335 31582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:15.083055 31582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:15.083199 31582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:15.083364 31582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:15.093806 31582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:15.095227 31582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:15.096738 31582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:15.110329 31588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:15.110435 31598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:15.110342 31592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:15.111310 31582 server_base.cc:1034] running on GCE node
I20260810 23:38:15.111999 31582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:15.113020 31582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:15.114233 31582 hybrid_clock.cc:648] HybridClock initialized: now 1786405095114174 us; error 43 us; skew 500 ppm
I20260810 23:38:15.114568 31582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:15.117848 31582 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:15.119429 31582 webserver.cc:460] Webserver started at http://127.30.33.3:35587/ using document root <none> and password file <none>
I20260810 23:38:15.120208 31582 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:15.120337 31582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:15.120657 31582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:15.123876 31582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "0bde53b4ff9e43c5a639dba941cdb26b"
format_stamp: "Formatted at 2026-08-10 23:38:15 on dist-test-slave-pww0"
server_key: "a785e40166345087610c949e5dfd297a"
server_key_iv: "4484b8d87315333ea00f9ba62111d7fb"
server_key_version: "encryptionkey@0"
I20260810 23:38:15.124770 31582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "0bde53b4ff9e43c5a639dba941cdb26b"
format_stamp: "Formatted at 2026-08-10 23:38:15 on dist-test-slave-pww0"
server_key: "a785e40166345087610c949e5dfd297a"
server_key_iv: "4484b8d87315333ea00f9ba62111d7fb"
server_key_version: "encryptionkey@0"
I20260810 23:38:15.132285 31582 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260810 23:38:15.136162 31618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:15.137734 31582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:15.137913 31582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "0bde53b4ff9e43c5a639dba941cdb26b"
format_stamp: "Formatted at 2026-08-10 23:38:15 on dist-test-slave-pww0"
server_key: "a785e40166345087610c949e5dfd297a"
server_key_iv: "4484b8d87315333ea00f9ba62111d7fb"
server_key_version: "encryptionkey@0"
I20260810 23:38:15.138130 31582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:15.149545 31582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:15.154312 31582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:15.154635 31582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:15.155804 31582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:15.157321 31582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:15.157408 31582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:15.157534 31582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:15.157593 31582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:15.191054 31582 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:45667
I20260810 23:38:15.191138 31819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:45667 every 8 connection(s)
I20260810 23:38:15.192481 31582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:15.198359 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31582
I20260810 23:38:15.198571 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:15.199118 30852 external_mini_cluster.cc:1428] Setting key 8dafce2b4c1e7aad4b26beb477d70350
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:15.215771 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:15.195379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:52117 (local address 127.30.33.62:33915)
0810 23:38:15.195893 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:15.195898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:15.197021 (+ 1123us) server_negotiation.cc:407] Connection header received
0810 23:38:15.199445 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:15.199450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:15.199580 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:15.199683 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:15.201795 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.202967 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.204841 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.205285 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.205389 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:15.209244 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:15.209249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:15.209253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:15.209291 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:15.211987 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.212841 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.212845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.212848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.212981 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.213614 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.213619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.213621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.213938 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:15.214071 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:15.215105 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0810 23:38:15.215342 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
I20260810 23:38:15.217026 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:15.196058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33915 (local address 127.30.33.3:52117)
0810 23:38:15.196836 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:15.196883 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:15.199083 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:15.200009 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:15.200023 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:15.200696 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:15.201582 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:15.201614 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.203142 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.203147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:15.204653 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:15.204696 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.205487 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.205492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:15.205750 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:15.206435 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:15.206463 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:15.208948 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:15.212158 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:15.212167 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:15.212191 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:15.212725 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:15.213109 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:15.213113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:15.213116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:15.213365 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:15.214116 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:15.214134 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:15.214634 (+ 500us) client_negotiation.cc:769] Sending connection context
0810 23:38:15.214969 (+ 335us) client_negotiation.cc:240] Negotiation successful
0810 23:38:15.215639 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":311,"threads_started":1}
I20260810 23:38:15.218420 31820 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33915
I20260810 23:38:15.218750 31820 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:15.219548 31820 heartbeater.cc:511] Master 127.30.33.62:33915 requested a full tablet report, sending...
I20260810 23:38:15.221240 30953 ts_manager.cc:194] Registered new tserver with Master: 0bde53b4ff9e43c5a639dba941cdb26b (127.30.33.3:45667)
I20260810 23:38:15.222077 30953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:52117
I20260810 23:38:15.229650 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:15.283517 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:15.267059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.30.33.1:36683)
0810 23:38:15.267819 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:15.267827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:15.267863 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:38:15.268059 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:15.268068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:15.268339 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:15.268491 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:15.269962 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.271031 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.272486 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.273197 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.273308 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:15.277694 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:15.277710 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:15.277759 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:15.277809 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:15.280765 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.281492 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.281499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.281501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.281631 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.281981 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.281984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.281986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.282345 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:15.282578 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:15.282900 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 23:38:15.283111 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":413,"threads_started":1}
W20260810 23:38:15.284924 31157 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53352
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:15.338215 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:15.322936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.30.33.1:36683)
0810 23:38:15.323167 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:15.323174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:15.323488 (+ 314us) server_negotiation.cc:407] Connection header received
0810 23:38:15.323713 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:15.323738 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:15.323846 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:15.323935 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:15.325302 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.326000 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.327383 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.327894 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.327997 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:15.332801 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:15.332807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:15.332810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:15.332855 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:15.335602 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.336453 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.336457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.336460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.336604 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.336955 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.336957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.336959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.337208 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:15.337393 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:15.337702 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 23:38:15.337910 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:38:15.339761 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31066
I20260810 23:38:15.353183 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:15.340690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.30.33.1:36683)
0810 23:38:15.340897 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:15.340904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:15.341170 (+ 266us) server_negotiation.cc:407] Connection header received
0810 23:38:15.341319 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:15.341322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:15.341380 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:15.341468 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:15.342643 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.343352 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.344698 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.345156 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.345275 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:15.347631 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:15.347637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:15.347640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:15.347679 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:15.350363 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.351193 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.351196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.351199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.351351 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.351853 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.351857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.351859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.352144 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:15.352352 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:15.352619 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 23:38:15.352824 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:38:15.353932 31315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:15.438905 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31066
I20260810 23:38:15.466446 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31340
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:38:15.484805 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:15.467555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.30.33.2:38493)
0810 23:38:15.468231 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:15.468238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:15.468264 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:38:15.468338 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:15.468344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:15.468555 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:15.468692 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:15.470698 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.471891 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.473740 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.474356 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.474466 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:15.478837 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:15.478853 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:15.478869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:15.478919 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:15.481869 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.482587 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.482596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.482598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.482756 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.483169 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.483172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.483174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.483621 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:15.483956 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:15.484229 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 23:38:15.484468 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":320,"threads_started":1}
I20260810 23:38:15.486649 31565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:15.570663 31572 heartbeater.cc:503] Master 127.30.33.62:33915 was elected leader, sending a full tablet report...
I20260810 23:38:15.572356 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31340
I20260810 23:38:15.597420 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31582
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:38:15.614609 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:15.598989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.30.33.3:45667)
0810 23:38:15.599871 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:15.599879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:15.599910 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:38:15.599976 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:15.599982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:15.600205 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:15.600341 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:15.601621 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.602686 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.604088 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:15.604672 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:15.604792 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:15.609035 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:15.609052 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:15.609070 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:15.609112 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:15.611906 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.612587 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.612595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.612598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.612754 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:15.613088 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:15.613092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:15.613094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:15.613501 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:15.613770 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:15.614027 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:38:15.614246 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":558,"threads_started":1}
I20260810 23:38:15.616295 31815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:15.700997 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31582
I20260810 23:38:15.727061 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 30881
I20260810 23:38:15.728632 31011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:15.836550 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 30881
2026-08-10T23:38:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4750 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[31858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[31858](info): set up 2 sockets
Aug 10 23:38:15 dist-test-slave-pww0 krb5kdc[31858](info): commencing operation
krb5kdc: starting...
W20260810 23:38:17.110983 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:17 dist-test-slave-pww0 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:17.331445 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:43773
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:17.673782 31882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:17.674104 31882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:17.674198 31882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:17.681788 31882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:17.681922 31882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:17.681970 31882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:17.682029 31882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:17.682075 31882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:17.692480 31882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:43773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:17.693856 31882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:17.695195 31882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:17.707484 31894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:17.707441 31899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:17.707528 31893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:17.708308 31882 server_base.cc:1034] running on GCE node
I20260810 23:38:17.709158 31882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:17.710690 31882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:17.711920 31882 hybrid_clock.cc:648] HybridClock initialized: now 1786405097711852 us; error 42 us; skew 500 ppm
I20260810 23:38:17.712303 31882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:17 dist-test-slave-pww0 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405097, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:17.715518 31882 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:17.717232 31882 webserver.cc:460] Webserver started at http://127.30.33.62:40405/ using document root <none> and password file <none>
I20260810 23:38:17.717967 31882 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:17.718083 31882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:17.718412 31882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:17.721627 31882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "92e02456660b4e1d8e1de965135e898c"
format_stamp: "Formatted at 2026-08-10 23:38:17 on dist-test-slave-pww0"
server_key: "38c7cebbf7ef0331d700dc58cb86ed98"
server_key_iv: "751a73592f713997349a5376e547c0d8"
server_key_version: "encryptionkey@0"
I20260810 23:38:17.722435 31882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "92e02456660b4e1d8e1de965135e898c"
format_stamp: "Formatted at 2026-08-10 23:38:17 on dist-test-slave-pww0"
server_key: "38c7cebbf7ef0331d700dc58cb86ed98"
server_key_iv: "751a73592f713997349a5376e547c0d8"
server_key_version: "encryptionkey@0"
I20260810 23:38:17.728034 31882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:38:17.731774 31915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:17.733340 31882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:17.733543 31882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "92e02456660b4e1d8e1de965135e898c"
format_stamp: "Formatted at 2026-08-10 23:38:17 on dist-test-slave-pww0"
server_key: "38c7cebbf7ef0331d700dc58cb86ed98"
server_key_iv: "751a73592f713997349a5376e547c0d8"
server_key_version: "encryptionkey@0"
I20260810 23:38:17.733760 31882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:17.750062 31882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:17.754781 31882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:17.755095 31882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:17.774384 31882 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:43773
I20260810 23:38:17.774439 32000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:43773 every 8 connection(s)
I20260810 23:38:17.775933 31882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:17.780022 32001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:17.783695 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31882
I20260810 23:38:17.783941 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:17.784790 30852 external_mini_cluster.cc:1428] Setting key 12ede491ddc5291bfd2af672e1acc7b2
I20260810 23:38:17.794132 32001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c: Bootstrap starting.
I20260810 23:38:17.798308 32001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:17.799840 32001 log.cc:824] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:38:17 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:17.803337 32001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c: No bootstrap required, opened a new log
I20260810 23:38:17.810353 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:17.787089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.30.33.62:43773)
0810 23:38:17.787970 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:17.787983 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:17.788028 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:38:17.789779 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:17.789807 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:17.790309 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:17.790690 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:17.792662 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:17.794057 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:17.795850 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:17.796638 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:17.796797 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:17.801875 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:17.801885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:17.801904 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:17.801947 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:17.805218 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:17.806215 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:17.806226 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:17.806234 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:17.806436 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:17.806855 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:17.806858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:17.806861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:17.807741 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:17.808081 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:17.808670 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 23:38:17.809175 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":399,"threads_started":1}
I20260810 23:38:17.812654 32001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e02456660b4e1d8e1de965135e898c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43773 } }
I20260810 23:38:17.813053 32001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:17.813179 32001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e02456660b4e1d8e1de965135e898c, State: Initialized, Role: FOLLOWER
I20260810 23:38:17.813827 32001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e02456660b4e1d8e1de965135e898c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43773 } }
I20260810 23:38:17.814067 32001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:17.814172 32001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:17.814302 32001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:17.816571 32001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e02456660b4e1d8e1de965135e898c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43773 } }
I20260810 23:38:17.817072 32001 leader_election.cc:302] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e02456660b4e1d8e1de965135e898c; no voters:
I20260810 23:38:17.817639 32001 leader_election.cc:288] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:17.817813 32008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:17.818514 32008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [term 1 LEADER]: Becoming Leader. State: Replica: 92e02456660b4e1d8e1de965135e898c, State: Running, Role: LEADER
I20260810 23:38:17.819149 32008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e02456660b4e1d8e1de965135e898c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43773 } }
I20260810 23:38:17.819767 32001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:17.822346 32009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92e02456660b4e1d8e1de965135e898c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e02456660b4e1d8e1de965135e898c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43773 } } }
I20260810 23:38:17.822629 32010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e02456660b4e1d8e1de965135e898c. Latest consensus state: current_term: 1 leader_uuid: "92e02456660b4e1d8e1de965135e898c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e02456660b4e1d8e1de965135e898c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43773 } } }
I20260810 23:38:17.822918 32010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:17.822918 32009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:17.823911 32022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:17.829444 32022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:17.839751 32022 catalog_manager.cc:1347] Generated new cluster ID: afeb83a4fe1748629dad778211444fac
I20260810 23:38:17.839901 32022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:17.859872 32022 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:17.861872 32022 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:17.871078 32022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92e02456660b4e1d8e1de965135e898c: Generated new TSK 0
I20260810 23:38:17.872043 32022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:17.938702 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43773
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:18.312032 32041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:18.312454 32041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:18.312572 32041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:18.320188 32041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:18.320322 32041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:18.320458 32041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:18.330873 32041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:18.332276 32041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:18.333621 32041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:18.347427 32051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:18.347564 32055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:18.347569 32062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:18.348521 32041 server_base.cc:1034] running on GCE node
I20260810 23:38:18.349207 32041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:18.350297 32041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:18.351603 32041 hybrid_clock.cc:648] HybridClock initialized: now 1786405098351533 us; error 58 us; skew 500 ppm
I20260810 23:38:18.351985 32041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:18 dist-test-slave-pww0 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405098, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:18.360585 32041 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:18.362404 32041 webserver.cc:460] Webserver started at http://127.30.33.1:38411/ using document root <none> and password file <none>
I20260810 23:38:18.363166 32041 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:18.363281 32041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:18.363647 32041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:18.367005 32041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8"
format_stamp: "Formatted at 2026-08-10 23:38:18 on dist-test-slave-pww0"
server_key: "98f0c157f718913afb7b1c5c6ae55d03"
server_key_iv: "727b91fce601ca998b26a00141dd147d"
server_key_version: "encryptionkey@0"
I20260810 23:38:18.367988 32041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8"
format_stamp: "Formatted at 2026-08-10 23:38:18 on dist-test-slave-pww0"
server_key: "98f0c157f718913afb7b1c5c6ae55d03"
server_key_iv: "727b91fce601ca998b26a00141dd147d"
server_key_version: "encryptionkey@0"
I20260810 23:38:18.374342 32041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260810 23:38:18.378371 32074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:18.379995 32041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:18.380172 32041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8"
format_stamp: "Formatted at 2026-08-10 23:38:18 on dist-test-slave-pww0"
server_key: "98f0c157f718913afb7b1c5c6ae55d03"
server_key_iv: "727b91fce601ca998b26a00141dd147d"
server_key_version: "encryptionkey@0"
I20260810 23:38:18.380388 32041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:18.398236 32041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:18.402588 32041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:18.402881 32041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:18.403968 32041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:18.405411 32041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:18.405488 32041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:18.405654 32041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:18.405710 32041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:18.440049 32041 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:41123
I20260810 23:38:18.440092 32263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:41123 every 8 connection(s)
I20260810 23:38:18.441421 32041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:18.448436 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 32041
I20260810 23:38:18.448652 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:18.449288 30852 external_mini_cluster.cc:1428] Setting key b2daeb7ddd32bb10d151367640cf7729
Aug 10 23:38:18 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405098, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:18.463505 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:18.444121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:41837 (local address 127.30.33.62:43773)
0810 23:38:18.444564 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:18.444568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:18.445641 (+ 1073us) server_negotiation.cc:407] Connection header received
0810 23:38:18.447652 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:18.447659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:18.447761 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:18.447870 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:18.449717 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:18.450617 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:18.452459 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:18.452906 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:18.453047 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:18.457103 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:18.457109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:18.457113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:18.457157 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:18.460028 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:18.460919 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:18.460924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:18.460926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:18.461065 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:18.461562 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:18.461567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:18.461569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:18.461868 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:18.462013 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:18.462840 (+ 827us) server_negotiation.cc:299] Negotiation successful
0810 23:38:18.463106 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260810 23:38:18.464567 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:18.444643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43773 (local address 127.30.33.1:41837)
0810 23:38:18.445445 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:18.445492 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:18.447316 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:18.448163 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:18.448173 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:18.448842 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:18.449515 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:18.449546 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:18.450797 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:18.450801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:18.452301 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:18.452318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:18.453170 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:18.453177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:18.453422 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:18.454126 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:18.454155 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:18.456794 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:18.460228 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:18.460237 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:18.460254 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:18.460764 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:18.461193 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:18.461198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:18.461200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:18.461431 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:18.462008 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:18.462023 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:18.462415 (+ 392us) client_negotiation.cc:769] Sending connection context
0810 23:38:18.462684 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 23:38:18.463296 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":332,"threads_started":1}
I20260810 23:38:18.466490 32266 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43773
I20260810 23:38:18.466823 32266 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:18.467716 32266 heartbeater.cc:511] Master 127.30.33.62:43773 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
I20260810 23:38:18.470319 31944 ts_manager.cc:194] Registered new tserver with Master: 64cc4d8d04e84051a7a27aa75fb9c6d8 (127.30.33.1:41123)
I20260810 23:38:18.471968 31944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:41837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:18.499804 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43773
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:18.847136 32276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:18.847498 32276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:18.847656 32276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:18.855149 32276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:18.855288 32276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:18.855440 32276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:18.865619 32276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:18.867089 32276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:18.868593 32276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:18.882062 32287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:18.882077 32286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:18.882112 32293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:18.883307 32276 server_base.cc:1034] running on GCE node
I20260810 23:38:18.883998 32276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:18.885083 32276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:18.886343 32276 hybrid_clock.cc:648] HybridClock initialized: now 1786405098886279 us; error 45 us; skew 500 ppm
I20260810 23:38:18.886716 32276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:18 dist-test-slave-pww0 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405098, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:18.890183 32276 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:18.891898 32276 webserver.cc:460] Webserver started at http://127.30.33.2:36105/ using document root <none> and password file <none>
I20260810 23:38:18.892645 32276 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:18.892756 32276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:18.893059 32276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:18.896292 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "6f1a39c9a7ae400db7226c6ff1a691b8"
format_stamp: "Formatted at 2026-08-10 23:38:18 on dist-test-slave-pww0"
server_key: "73f8faefbd9ccb3302289d89fad59bf2"
server_key_iv: "684a2e383b037c9c008bca04dfed7ce9"
server_key_version: "encryptionkey@0"
I20260810 23:38:18.897101 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "6f1a39c9a7ae400db7226c6ff1a691b8"
format_stamp: "Formatted at 2026-08-10 23:38:18 on dist-test-slave-pww0"
server_key: "73f8faefbd9ccb3302289d89fad59bf2"
server_key_iv: "684a2e383b037c9c008bca04dfed7ce9"
server_key_version: "encryptionkey@0"
I20260810 23:38:18.902742 32276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:38:18.906826 32301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:18.908702 32276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:38:18.908932 32276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "6f1a39c9a7ae400db7226c6ff1a691b8"
format_stamp: "Formatted at 2026-08-10 23:38:18 on dist-test-slave-pww0"
server_key: "73f8faefbd9ccb3302289d89fad59bf2"
server_key_iv: "684a2e383b037c9c008bca04dfed7ce9"
server_key_version: "encryptionkey@0"
I20260810 23:38:18.909184 32276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:18.950781 32276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:18.955217 32276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:18.955567 32276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:18.956668 32276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:18.958206 32276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:18.958292 32276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:18.958431 32276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:18.958487 32276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:18.990983 32276 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:36795
I20260810 23:38:18.991040 32516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:36795 every 8 connection(s)
I20260810 23:38:18.992412 32276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:18.992837 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 32276
I20260810 23:38:18.993026 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:18.993604 30852 external_mini_cluster.cc:1428] Setting key 59d2d0c597b6e1192802b7a3d0ffb1d8
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405098, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
I20260810 23:38:19.016238 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:18.995139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:46807 (local address 127.30.33.62:43773)
0810 23:38:18.995639 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:18.995645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:18.996942 (+ 1297us) server_negotiation.cc:407] Connection header received
0810 23:38:18.999575 (+ 2633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:18.999579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:18.999660 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:18.999779 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:19.002007 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.002798 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.004890 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.005429 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.005564 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.009446 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:19.009453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:19.009456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:19.009498 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:19.012616 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.013590 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.013595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.013597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.013778 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.014325 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.014329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.014331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.014631 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:19.014782 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.015604 (+ 822us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.015865 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260810 23:38:19.017446 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:18.995882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43773 (local address 127.30.33.2:46807)
0810 23:38:18.996760 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:18.996814 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:18.999168 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:19.000099 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:19.000111 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:19.000911 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:19.001793 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.001820 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.002986 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.002991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.004651 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.004669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.005674 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.005679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.005924 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.006638 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:19.006664 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:19.009127 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:19.012812 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:19.012823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:19.012841 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:19.013427 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:19.013903 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:19.013907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:19.013909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:19.014147 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:19.014779 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:19.014788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:19.015159 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 23:38:19.015426 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 23:38:19.016031 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":335,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
I20260810 23:38:19.018965 32517 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43773
I20260810 23:38:19.019289 32517 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:19.020076 32517 heartbeater.cc:511] Master 127.30.33.62:43773 requested a full tablet report, sending...
I20260810 23:38:19.021610 31944 ts_manager.cc:194] Registered new tserver with Master: 6f1a39c9a7ae400db7226c6ff1a691b8 (127.30.33.2:36795)
I20260810 23:38:19.022476 31944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:46807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:19.042876 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43773
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:19.391551 32523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:19.391894 32523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:19.392035 32523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:19.399556 32523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:19.399694 32523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:19.399854 32523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:19.409967 32523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:19.411391 32523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:19.412824 32523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:19.426472 32535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:19.426527 32534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:19.426559 32538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:19.427136 32523 server_base.cc:1034] running on GCE node
I20260810 23:38:19.427901 32523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:19.428893 32523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:19.430131 32523 hybrid_clock.cc:648] HybridClock initialized: now 1786405099430074 us; error 35 us; skew 500 ppm
I20260810 23:38:19.430500 32523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:19.433974 32523 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:19.435662 32523 webserver.cc:460] Webserver started at http://127.30.33.3:40749/ using document root <none> and password file <none>
I20260810 23:38:19.436389 32523 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:19.436509 32523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:19.436820 32523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:19.440717 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "1e0654147a5648da814e5ff4c32612ea"
format_stamp: "Formatted at 2026-08-10 23:38:19 on dist-test-slave-pww0"
server_key: "a90e43e239e22f8053371af30ff3202d"
server_key_iv: "89e697ba7dae41f52d3ab0e74695cd35"
server_key_version: "encryptionkey@0"
I20260810 23:38:19.441893 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "1e0654147a5648da814e5ff4c32612ea"
format_stamp: "Formatted at 2026-08-10 23:38:19 on dist-test-slave-pww0"
server_key: "a90e43e239e22f8053371af30ff3202d"
server_key_iv: "89e697ba7dae41f52d3ab0e74695cd35"
server_key_version: "encryptionkey@0"
I20260810 23:38:19.449169 32523 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260810 23:38:19.453406 32547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:19.455112 32523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 23:38:19.455325 32523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "1e0654147a5648da814e5ff4c32612ea"
format_stamp: "Formatted at 2026-08-10 23:38:19 on dist-test-slave-pww0"
server_key: "a90e43e239e22f8053371af30ff3202d"
server_key_iv: "89e697ba7dae41f52d3ab0e74695cd35"
server_key_version: "encryptionkey@0"
I20260810 23:38:19.455600 32523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:19.476451 32266 heartbeater.cc:503] Master 127.30.33.62:43773 was elected leader, sending a full tablet report...
I20260810 23:38:19.480448 32523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:19.485037 32523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:19.485335 32523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:19.486428 32523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:19.487916 32523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:19.487998 32523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:19.488138 32523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:19.488193 32523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:19.520900 32523 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:46815
I20260810 23:38:19.520956 323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:46815 every 8 connection(s)
I20260810 23:38:19.522246 32523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:19.526259 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 32523
I20260810 23:38:19.526476 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:19.526952 30852 external_mini_cluster.cc:1428] Setting key 832469c813c805aa791d30d925d90a07
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:19.544288 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.524884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:38361 (local address 127.30.33.62:43773)
0810 23:38:19.525375 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.525380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.526427 (+ 1047us) server_negotiation.cc:407] Connection header received
0810 23:38:19.528812 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.528816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.528891 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.529002 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:19.531010 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.531884 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.533498 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.533887 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.533977 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.537652 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:19.537658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:19.537661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:19.537699 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:19.540432 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.541364 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.541371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.541374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.541527 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.542055 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.542061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.542065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.542481 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:19.542679 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.543615 (+ 936us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.543875 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":230,"threads_started":1}
I20260810 23:38:19.545336 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.525498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43773 (local address 127.30.33.3:38361)
0810 23:38:19.526270 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:19.526311 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:19.528500 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:19.529303 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:19.529318 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:19.530132 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:19.530834 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.530860 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.532077 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.532081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.533380 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.533394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.534118 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.534124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.534340 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.534958 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:19.534980 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:19.537386 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:19.540611 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:19.540619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:19.540634 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:19.541242 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:19.541640 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:19.541643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:19.541647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:19.541911 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:19.542666 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:19.542675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:19.543163 (+ 488us) client_negotiation.cc:769] Sending connection context
0810 23:38:19.543501 (+ 338us) client_negotiation.cc:240] Negotiation successful
0810 23:38:19.544080 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":308,"threads_started":1}
I20260810 23:38:19.546957 325 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43773
I20260810 23:38:19.547273 325 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:19.548029 325 heartbeater.cc:511] Master 127.30.33.62:43773 requested a full tablet report, sending...
I20260810 23:38:19.549546 31944 ts_manager.cc:194] Registered new tserver with Master: 1e0654147a5648da814e5ff4c32612ea (127.30.33.3:46815)
I20260810 23:38:19.550379 31944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:38361
I20260810 23:38:19.555737 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:38:19.575562 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.562344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.30.33.62:43773)
0810 23:38:19.562522 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.562526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.563039 (+ 513us) server_negotiation.cc:407] Connection header received
0810 23:38:19.563200 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.563204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.563282 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.563379 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:19.564912 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.565675 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.566924 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.567333 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.567427 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.569857 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:19.569862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:19.569865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:19.569901 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:19.572539 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.573322 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.573327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.573329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.573491 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.573998 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.574002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.574005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.574504 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:19.574719 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.575075 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.575277 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:38:19.582576 31944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:19.590443 31944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:19.625644 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.604775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:36795 (local address 127.0.0.1:58526)
0810 23:38:19.615221 (+ 10446us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:19.615249 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:19.615376 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:19.616818 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:19.616828 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:19.616873 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:19.617379 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.617396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.619847 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.619853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.622517 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.622537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.624519 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.624526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.624741 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.624791 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 23:38:19.624924 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 23:38:19.625142 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10316,"mutex_wait_us":164,"thread_start_us":9139,"threads_started":1}
I20260810 23:38:19.626662 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.605065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.30.33.2:36795)
0810 23:38:19.615952 (+ 10887us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.615964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.616004 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:38:19.616131 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.616139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.616494 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.616739 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:19.617751 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.619666 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.622733 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.624306 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.624436 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.624671 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.626161 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.626366 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10686,"thread_start_us":10576,"threads_started":1}
I20260810 23:38:19.626718 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.604476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.30.33.3:46815)
0810 23:38:19.617558 (+ 13082us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.617567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.617600 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:38:19.617700 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.617707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.617964 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.618174 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:19.618832 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.620471 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.623167 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.624527 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.624659 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.624928 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.626159 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.626351 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12878,"thread_start_us":12668,"threads_started":1}
I20260810 23:38:19.627033 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.615816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.30.33.1:41123)
0810 23:38:19.616533 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.616543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.618015 (+ 1472us) server_negotiation.cc:407] Connection header received
0810 23:38:19.618761 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.618768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.619015 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.619217 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:19.619866 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.621370 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.624079 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.625601 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.625732 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.625964 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.626521 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.626708 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":329,"threads_started":1}
I20260810 23:38:19.626003 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.604206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:46815 (local address 127.0.0.1:43098)
0810 23:38:19.614375 (+ 10169us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:19.614430 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:19.614667 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:19.618175 (+ 3508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:19.618180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:19.618207 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:19.618661 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.618674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.620681 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.620686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.623003 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.623015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.625085 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.625091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.625268 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.625309 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:38:19.625414 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:38:19.625563 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9873,"mutex_wait_us":31,"thread_start_us":10735,"threads_started":1}
I20260810 23:38:19.626901 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.615644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:41123 (local address 127.0.0.1:34916)
0810 23:38:19.617836 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:19.617856 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:19.618544 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:19.619228 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:19.619233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:19.619260 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:19.619632 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.619644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.621544 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.621549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.623883 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.623899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.626116 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.626122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.626302 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.626336 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:38:19.626424 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:38:19.626560 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"mutex_wait_us":56,"thread_start_us":1886,"threads_started":1}
I20260810 23:38:19.630976 32158 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc4467d9dcf4ce592a12aa46c731ac4 (DEFAULT_TABLE table=test-table [id=d2b06baea6d34ba68a48137396fd1066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:19.631198 32386 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc4467d9dcf4ce592a12aa46c731ac4 (DEFAULT_TABLE table=test-table [id=d2b06baea6d34ba68a48137396fd1066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:19.631207 32667 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc4467d9dcf4ce592a12aa46c731ac4 (DEFAULT_TABLE table=test-table [id=d2b06baea6d34ba68a48137396fd1066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:19.632650 32158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc4467d9dcf4ce592a12aa46c731ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:19.632872 32386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc4467d9dcf4ce592a12aa46c731ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:19.633504 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc4467d9dcf4ce592a12aa46c731ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:19.645272 355 tablet_bootstrap.cc:492] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8: Bootstrap starting.
I20260810 23:38:19.649621 355 tablet_bootstrap.cc:654] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:19.650872 355 log.cc:824] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:19.652082 359 tablet_bootstrap.cc:492] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8: Bootstrap starting.
I20260810 23:38:19.653167 357 tablet_bootstrap.cc:492] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea: Bootstrap starting.
I20260810 23:38:19.656232 355 tablet_bootstrap.cc:492] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8: No bootstrap required, opened a new log
I20260810 23:38:19.656533 355 ts_tablet_manager.cc:1397] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260810 23:38:19.657364 359 tablet_bootstrap.cc:654] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:19.657351 357 tablet_bootstrap.cc:654] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:19.658825 357 log.cc:824] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:19.658804 359 log.cc:824] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:19.662058 359 tablet_bootstrap.cc:492] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8: No bootstrap required, opened a new log
I20260810 23:38:19.662341 359 ts_tablet_manager.cc:1397] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 23:38:19.662359 357 tablet_bootstrap.cc:492] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea: No bootstrap required, opened a new log
I20260810 23:38:19.662637 357 ts_tablet_manager.cc:1397] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260810 23:38:19.666805 355 raft_consensus.cc:348] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.667351 355 raft_consensus.cc:374] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:19.667539 355 raft_consensus.cc:729] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cc4d8d04e84051a7a27aa75fb9c6d8, State: Initialized, Role: FOLLOWER
I20260810 23:38:19.668313 355 consensus_queue.cc:260] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.670367 355 ts_tablet_manager.cc:1428] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260810 23:38:19.671233 359 raft_consensus.cc:348] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.671761 359 raft_consensus.cc:374] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:19.671871 359 raft_consensus.cc:729] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1a39c9a7ae400db7226c6ff1a691b8, State: Initialized, Role: FOLLOWER
I20260810 23:38:19.672485 359 consensus_queue.cc:260] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.672880 357 raft_consensus.cc:348] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.673410 357 raft_consensus.cc:374] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:19.673543 357 raft_consensus.cc:729] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0654147a5648da814e5ff4c32612ea, State: Initialized, Role: FOLLOWER
I20260810 23:38:19.673677 32517 heartbeater.cc:503] Master 127.30.33.62:43773 was elected leader, sending a full tablet report...
I20260810 23:38:19.674392 359 ts_tablet_manager.cc:1428] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260810 23:38:19.674240 357 consensus_queue.cc:260] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.675418 325 heartbeater.cc:503] Master 127.30.33.62:43773 was elected leader, sending a full tablet report...
I20260810 23:38:19.676124 357 ts_tablet_manager.cc:1428] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260810 23:38:19.689872 370 raft_consensus.cc:482] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:19.690231 370 raft_consensus.cc:504] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.692245 370 leader_election.cc:288] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e0654147a5648da814e5ff4c32612ea (127.30.33.3:46815), 6f1a39c9a7ae400db7226c6ff1a691b8 (127.30.33.2:36795)
W20260810 23:38:19.693931 32267 tablet.cc:2369] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:38:19.699361 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.692626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:34975 (local address 127.30.33.3:46815)
0810 23:38:19.692850 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.692857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.693588 (+ 731us) server_negotiation.cc:407] Connection header received
0810 23:38:19.693841 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.693848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.693923 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.694001 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:19.694769 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.695657 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.697635 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.698418 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.698534 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.698648 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.698981 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.699107 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:38:19.699404 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.692896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:42151 (local address 127.30.33.2:36795)
0810 23:38:19.693109 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.693115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.693670 (+ 555us) server_negotiation.cc:407] Connection header received
0810 23:38:19.693903 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.693909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.693992 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.694082 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:19.694893 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.695720 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.697599 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.698400 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.698548 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.698651 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.699022 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.699155 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:38:19.699544 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.692518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:36795 (local address 127.30.33.1:42151)
0810 23:38:19.693561 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:19.693575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:19.693708 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:19.694200 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:19.694204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:19.694239 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:19.694746 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.694758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.695864 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.695872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.697430 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.697443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.698704 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.698711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.698872 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.698895 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:38:19.698976 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:38:19.699115 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":190,"thread_start_us":368,"threads_started":1}
I20260810 23:38:19.699795 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.692508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:46815 (local address 127.30.33.1:34975)
0810 23:38:19.693477 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:19.693494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:19.693666 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:19.694041 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:19.694046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:19.694079 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:19.694593 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.694606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.695779 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.695785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.697421 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:19.697434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.698650 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.698659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:19.698813 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.698853 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:38:19.698936 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:38:19.699072 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":26,"thread_start_us":703,"threads_started":1}
I20260810 23:38:19.700655 32419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc4467d9dcf4ce592a12aa46c731ac4" candidate_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" is_pre_election: true
I20260810 23:38:19.700670 32697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc4467d9dcf4ce592a12aa46c731ac4" candidate_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0654147a5648da814e5ff4c32612ea" is_pre_election: true
I20260810 23:38:19.701139 32419 raft_consensus.cc:2454] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64cc4d8d04e84051a7a27aa75fb9c6d8 in term 0.
I20260810 23:38:19.701155 32697 raft_consensus.cc:2454] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64cc4d8d04e84051a7a27aa75fb9c6d8 in term 0.
I20260810 23:38:19.701800 32077 leader_election.cc:302] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64cc4d8d04e84051a7a27aa75fb9c6d8, 6f1a39c9a7ae400db7226c6ff1a691b8; no voters:
I20260810 23:38:19.702214 370 raft_consensus.cc:2781] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:19.702346 370 raft_consensus.cc:482] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:19.702461 370 raft_consensus.cc:3037] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:19.704864 370 raft_consensus.cc:504] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.705601 370 leader_election.cc:288] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e0654147a5648da814e5ff4c32612ea (127.30.33.3:46815), 6f1a39c9a7ae400db7226c6ff1a691b8 (127.30.33.2:36795)
I20260810 23:38:19.706140 32697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc4467d9dcf4ce592a12aa46c731ac4" candidate_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0654147a5648da814e5ff4c32612ea"
I20260810 23:38:19.706140 32419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc4467d9dcf4ce592a12aa46c731ac4" candidate_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8"
I20260810 23:38:19.706373 32419 raft_consensus.cc:3037] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:19.706377 32697 raft_consensus.cc:3037] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:19.709755 32697 raft_consensus.cc:2454] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64cc4d8d04e84051a7a27aa75fb9c6d8 in term 1.
I20260810 23:38:19.709755 32419 raft_consensus.cc:2454] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64cc4d8d04e84051a7a27aa75fb9c6d8 in term 1.
I20260810 23:38:19.710351 32081 leader_election.cc:302] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e0654147a5648da814e5ff4c32612ea, 64cc4d8d04e84051a7a27aa75fb9c6d8, 6f1a39c9a7ae400db7226c6ff1a691b8; no voters:
I20260810 23:38:19.710728 370 raft_consensus.cc:2781] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:19.711194 370 raft_consensus.cc:686] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [term 1 LEADER]: Becoming Leader. State: Replica: 64cc4d8d04e84051a7a27aa75fb9c6d8, State: Running, Role: LEADER
I20260810 23:38:19.711732 370 consensus_queue.cc:237] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } }
I20260810 23:38:19.716372 31944 catalog_manager.cc:5495] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 64cc4d8d04e84051a7a27aa75fb9c6d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64cc4d8d04e84051a7a27aa75fb9c6d8 (127.30.33.1). New cstate: current_term: 1 leader_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0654147a5648da814e5ff4c32612ea" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64cc4d8d04e84051a7a27aa75fb9c6d8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 41123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f1a39c9a7ae400db7226c6ff1a691b8" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36795 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:19.744872 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.728957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.30.33.1:41123)
0810 23:38:19.729156 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.729161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.729660 (+ 499us) server_negotiation.cc:407] Connection header received
0810 23:38:19.729862 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.729867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.729951 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.730038 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:19.731656 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.732394 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.733698 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.734109 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.734237 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.738492 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:19.738512 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:19.738530 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:19.738577 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:19.741854 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.742671 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.742680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.742683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.742866 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.743199 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.743202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.743205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.743642 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:19.743905 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.744287 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.744521 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 23:38:19.744899 32518 tablet.cc:2369] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 6f1a39c9a7ae400db7226c6ff1a691b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:38:19.774099 327 tablet.cc:2369] T 6cc4467d9dcf4ce592a12aa46c731ac4 P 1e0654147a5648da814e5ff4c32612ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:19.801877 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.786801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.30.33.1:41123)
0810 23:38:19.786975 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.786981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.787478 (+ 497us) server_negotiation.cc:407] Connection header received
0810 23:38:19.787683 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.787689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.787769 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.787857 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:19.789371 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.790114 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.791522 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.791965 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.792087 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.796144 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:19.796149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:19.796153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:19.796197 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:19.799033 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.799803 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.799809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.799812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.799960 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.800407 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.800411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.800413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.800714 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:19.801000 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.801298 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.801545 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:38:19 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:19.856710 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.841509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.30.33.1:41123)
0810 23:38:19.841700 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.841707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.842080 (+ 373us) server_negotiation.cc:407] Connection header received
0810 23:38:19.842280 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.842285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.842372 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.842465 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:19.844041 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.844831 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.846170 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.846622 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.846744 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.850835 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:19.850841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:19.850845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:19.850888 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:19.853703 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.854550 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.854555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.854558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.854725 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.855131 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.855135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.855137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.855453 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:19.855786 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.856094 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.856350 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 23:38:19.858728 32121 tablet_service.cc:2908] Rejecting scan request for tablet 6cc4467d9dcf4ce592a12aa46c731ac4: Uninitialized: clean time has not yet been initialized
I20260810 23:38:19.862663 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 32041
I20260810 23:38:19.876020 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:19.863687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.30.33.1:41123)
0810 23:38:19.863867 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:19.863871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:19.864251 (+ 380us) server_negotiation.cc:407] Connection header received
0810 23:38:19.864431 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:19.864435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:19.864511 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:19.864598 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:19.865840 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.866546 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.867913 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:19.868333 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:19.868428 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:19.870821 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:19.870827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:19.870830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:19.870865 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:19.873472 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.874180 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.874185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.874187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.874329 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:19.874733 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:19.874737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:19.874740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:19.875023 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:19.875226 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:19.875498 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 23:38:19.875748 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:38:19.876751 32257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:19.972065 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 32041
I20260810 23:38:20.000413 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 32276
Aug 10 23:38:20 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:38:20.016067 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:20.001157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.30.33.2:36795)
0810 23:38:20.001358 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:20.001364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:20.001433 (+ 69us) server_negotiation.cc:407] Connection header received
0810 23:38:20.001621 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:20.001627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:20.001737 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:20.001826 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:20.003047 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:20.003897 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:20.005237 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:20.005637 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:20.005752 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:20.009953 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:20.009970 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:20.009988 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:20.010027 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:20.013052 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:20.013812 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:20.013820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:20.013823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:20.013956 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:20.014374 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:20.014378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:20.014396 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:20.014832 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:20.015079 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:20.015547 (+ 468us) server_negotiation.cc:299] Negotiation successful
0810 23:38:20.015773 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:38:20.016772 32504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:20.106843 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 32276
I20260810 23:38:20.132514 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 32523
Aug 10 23:38:20 dist-test-slave-pww0 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:38:20.147727 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:20.133446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.30.33.3:46815)
0810 23:38:20.133629 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:20.133634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:20.133655 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:38:20.133810 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:20.133815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:20.133885 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:20.133979 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:20.135219 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:20.135994 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:20.137296 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:20.137762 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:20.137899 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:20.142207 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:20.142228 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:20.142246 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:20.142286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:20.145121 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:20.145862 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:20.145869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:20.145872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:20.146033 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:20.146370 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:20.146373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:20.146375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:20.146746 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:20.146983 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:20.147193 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 23:38:20.147381 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:38:20.148411 308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:20.236903 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 32523
I20260810 23:38:20.254189 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31882
I20260810 23:38:20.255582 31993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:20.368481 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31882
2026-08-10T23:38:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4534 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:20 dist-test-slave-pww0 krb5kdc[422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:20 dist-test-slave-pww0 krb5kdc[422](info): set up 2 sockets
Aug 10 23:38:20 dist-test-slave-pww0 krb5kdc[422](info): commencing operation
krb5kdc: starting...
W20260810 23:38:21.648301 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:21 dist-test-slave-pww0 krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:21.874799 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35735
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:35735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:22.220093 439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:22.220434 439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:22.220584 439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:22.228068 439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:22.228205 439 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:38:22.228268 439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:22.228354 439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:22.228422 439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:22.228511 439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:22.238893 439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:35735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:22.240309 439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:22.241773 439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:22.254199 453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:22.254158 450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:22.254249 448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:22.255769 439 server_base.cc:1034] running on GCE node
I20260810 23:38:22.256628 439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:22.258155 439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:22.259420 439 hybrid_clock.cc:648] HybridClock initialized: now 1786405102259349 us; error 36 us; skew 500 ppm
I20260810 23:38:22.259840 439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:22 dist-test-slave-pww0 krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405102, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:22.263299 439 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:22.265041 439 webserver.cc:460] Webserver started at http://127.30.33.62:35073/ using document root <none> and password file <none>
I20260810 23:38:22.265810 439 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:22.265918 439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:22.266264 439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:22.269506 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "1faef59025114e05b5044ec76f067da8"
format_stamp: "Formatted at 2026-08-10 23:38:22 on dist-test-slave-pww0"
server_key: "6212d8d2e80c980a7c29176ead441247"
server_key_iv: "268c548b818a7940d677eb69ee5dd01d"
server_key_version: "encryptionkey@0"
I20260810 23:38:22.270337 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "1faef59025114e05b5044ec76f067da8"
format_stamp: "Formatted at 2026-08-10 23:38:22 on dist-test-slave-pww0"
server_key: "6212d8d2e80c980a7c29176ead441247"
server_key_iv: "268c548b818a7940d677eb69ee5dd01d"
server_key_version: "encryptionkey@0"
I20260810 23:38:22.276087 439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:38:22.280277 469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:22.281810 439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:22.282002 439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "1faef59025114e05b5044ec76f067da8"
format_stamp: "Formatted at 2026-08-10 23:38:22 on dist-test-slave-pww0"
server_key: "6212d8d2e80c980a7c29176ead441247"
server_key_iv: "268c548b818a7940d677eb69ee5dd01d"
server_key_version: "encryptionkey@0"
I20260810 23:38:22.282239 439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:22.313751 439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:22.318413 439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:22.318718 439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:22.340812 439 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:35735
I20260810 23:38:22.340837 600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:35735 every 8 connection(s)
I20260810 23:38:22.342312 439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:22.346333 601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:22.348112 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 439
I20260810 23:38:22.348366 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:22.349191 30852 external_mini_cluster.cc:1428] Setting key 4838f2f8c226b22056033d44876e386d
I20260810 23:38:22.359529 601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8: Bootstrap starting.
I20260810 23:38:22.363658 601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8: Neither blocks nor log segments found. Creating new log.
Aug 10 23:38:22 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:22.364931 601 log.cc:824] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:22.368587 601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8: No bootstrap required, opened a new log
I20260810 23:38:22.374186 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:22.351442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.30.33.62:35735)
0810 23:38:22.352324 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:22.352341 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:22.352393 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:38:22.354206 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:22.354233 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:22.354639 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:22.354999 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:22.356828 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.358184 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.359956 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.360785 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.360930 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:22.365984 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:22.365997 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:22.366020 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:22.366075 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:22.369522 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:22.370504 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:22.370512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:22.370519 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:22.370740 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:22.371146 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:22.371153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:22.371155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:22.371838 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:22.372092 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:22.372549 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 23:38:22.372999 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":393,"threads_started":1}
I20260810 23:38:22.377679 601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faef59025114e05b5044ec76f067da8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35735 } }
I20260810 23:38:22.378104 601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:22.378216 601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1faef59025114e05b5044ec76f067da8, State: Initialized, Role: FOLLOWER
I20260810 23:38:22.378770 601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faef59025114e05b5044ec76f067da8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35735 } }
I20260810 23:38:22.379019 601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:22.379165 601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:22.379369 601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:22.381760 601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faef59025114e05b5044ec76f067da8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35735 } }
I20260810 23:38:22.382349 601 leader_election.cc:302] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1faef59025114e05b5044ec76f067da8; no voters:
I20260810 23:38:22.382934 601 leader_election.cc:288] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:22.383085 608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:22.384310 608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [term 1 LEADER]: Becoming Leader. State: Replica: 1faef59025114e05b5044ec76f067da8, State: Running, Role: LEADER
I20260810 23:38:22.384850 608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faef59025114e05b5044ec76f067da8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35735 } }
I20260810 23:38:22.385210 601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:22.387692 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1faef59025114e05b5044ec76f067da8. Latest consensus state: current_term: 1 leader_uuid: "1faef59025114e05b5044ec76f067da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faef59025114e05b5044ec76f067da8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35735 } } }
I20260810 23:38:22.388090 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:22.388464 610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1faef59025114e05b5044ec76f067da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1faef59025114e05b5044ec76f067da8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35735 } } }
I20260810 23:38:22.388721 610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:22.391122 623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:22.396314 623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:22.405625 623 catalog_manager.cc:1347] Generated new cluster ID: a32f5724d49b453d841cd7affe73e086
I20260810 23:38:22.405779 623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:22.424805 623 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:22.426651 623 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:22.437083 623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8: Generated new TSK 0
I20260810 23:38:22.438241 623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:22.502998 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35735
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:22.855425 649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:22.855800 649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:22.855947 649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:22.864943 649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:22.865100 649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:22.865221 649 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:38:22.865334 649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:22.876784 649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:22.878216 649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:22.879808 649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:22.893954 659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:22.894001 669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:22.894022 661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:22.894855 649 server_base.cc:1034] running on GCE node
I20260810 23:38:22.895510 649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:22.896490 649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:22.897715 649 hybrid_clock.cc:648] HybridClock initialized: now 1786405102897639 us; error 44 us; skew 500 ppm
I20260810 23:38:22.898087 649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:22 dist-test-slave-pww0 krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405102, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:22.901345 649 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:22.902884 649 webserver.cc:460] Webserver started at http://127.30.33.1:39237/ using document root <none> and password file <none>
I20260810 23:38:22.903663 649 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:22.903769 649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:22.904065 649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:22.907078 649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "0996546142b44c76a3016eb5d68abc65"
format_stamp: "Formatted at 2026-08-10 23:38:22 on dist-test-slave-pww0"
server_key: "8869a32d9334d5fe421cbded08cd601f"
server_key_iv: "c327bba618b7ebaf02d5e03a61c16a18"
server_key_version: "encryptionkey@0"
I20260810 23:38:22.907927 649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "0996546142b44c76a3016eb5d68abc65"
format_stamp: "Formatted at 2026-08-10 23:38:22 on dist-test-slave-pww0"
server_key: "8869a32d9334d5fe421cbded08cd601f"
server_key_iv: "c327bba618b7ebaf02d5e03a61c16a18"
server_key_version: "encryptionkey@0"
I20260810 23:38:22.913287 649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 23:38:22.916955 685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:22.918398 649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:22.918560 649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "0996546142b44c76a3016eb5d68abc65"
format_stamp: "Formatted at 2026-08-10 23:38:22 on dist-test-slave-pww0"
server_key: "8869a32d9334d5fe421cbded08cd601f"
server_key_iv: "c327bba618b7ebaf02d5e03a61c16a18"
server_key_version: "encryptionkey@0"
I20260810 23:38:22.918803 649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:22.948410 649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:22.952800 649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:22.953096 649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:22.956143 649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:22.956241 649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:22.956396 649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:22.956457 649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:38:22 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405102, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:22.982676 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:22.956934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:57471 (local address 127.30.33.62:35735)
0810 23:38:22.957503 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:22.957510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:22.958744 (+ 1234us) server_negotiation.cc:407] Connection header received
0810 23:38:22.961709 (+ 2965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:22.961716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:22.961816 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:22.961917 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:22.965006 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.965898 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.968425 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.968997 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.969143 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:22.974871 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:22.974879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:22.974884 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:22.975047 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:22.978217 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:22.979604 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:22.979612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:22.979616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:22.979827 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:22.980613 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:22.980618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:22.980620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:22.980948 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:22.981089 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:22.982182 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0810 23:38:22.982392 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":313,"threads_started":1}
I20260810 23:38:22.984670 699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:22.957577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35735 (local address 127.30.33.1:57471)
0810 23:38:22.958522 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:22.958593 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:22.961304 (+ 2711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:22.962598 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:22.962615 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:22.963643 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:22.964757 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:22.964794 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.966135 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.966141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:22.968237 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:22.968259 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.969512 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.969517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:22.969902 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:22.970772 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:22.970796 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:22.974516 (+ 3720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:22.978385 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:22.978399 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:22.978421 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:22.979387 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:22.980014 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:22.980024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:22.980029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:22.980412 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:22.981081 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:22.981091 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:22.981668 (+ 577us) client_negotiation.cc:769] Sending connection context
0810 23:38:22.982018 (+ 350us) client_negotiation.cc:240] Negotiation successful
0810 23:38:22.982689 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":368,"threads_started":1}
I20260810 23:38:23.006659 858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:22.991639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35735 (local address 127.30.33.1:35629)
0810 23:38:22.992373 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:22.992398 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:22.992533 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:22.992921 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:22.992925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:22.993380 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:22.993702 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:22.993713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.995150 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.995154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:22.996633 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:22.996651 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.997916 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.997922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:22.998071 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:22.998706 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:22.998736 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:22.999734 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:23.003746 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:23.003752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:23.003756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:23.004362 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:23.004806 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:23.004811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:23.004816 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:23.004949 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:23.005506 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:23.005511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:23.005750 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 23:38:23.005926 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 23:38:23.006265 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":419,"threads_started":1}
I20260810 23:38:23.006814 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:22.991644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:35629 (local address 127.30.33.62:35735)
0810 23:38:22.991840 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:22.991846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:22.992501 (+ 655us) server_negotiation.cc:407] Connection header received
0810 23:38:22.992647 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:22.992653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:22.992752 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:22.992871 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:22.993841 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.995025 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.996855 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:22.997455 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:22.997595 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:22.999956 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:22.999962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:22.999965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:23.000012 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:23.003569 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:23.004537 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:23.004541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:23.004544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:23.004681 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:23.005067 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:23.005071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:23.005073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:23.005376 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:23.005531 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:23.006113 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 23:38:23.006427 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:38:23.008180 649 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43097
I20260810 23:38:23.008208 914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43097 every 8 connection(s)
I20260810 23:38:23.009524 649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:23.011708 915 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35735
I20260810 23:38:23.011936 915 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:23.012590 915 heartbeater.cc:511] Master 127.30.33.62:35735 requested a full tablet report, sending...
I20260810 23:38:23.014608 500 ts_manager.cc:194] Registered new tserver with Master: 0996546142b44c76a3016eb5d68abc65 (127.30.33.1:43097)
I20260810 23:38:23.016208 500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:57471
I20260810 23:38:23.016914 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 649
I20260810 23:38:23.017163 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:23.017917 30852 external_mini_cluster.cc:1428] Setting key a2438907b91effd4683697c722e74a35
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:23.067308 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35735
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:23.420042 922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:23.420346 922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:23.420483 922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:23.428017 922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:23.428150 922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:23.428280 922 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:38:23.428378 922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:23.438582 922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:23.440047 922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:23.441509 922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:23.456171 929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:23.456235 936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:23.456213 931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:23.456957 922 server_base.cc:1034] running on GCE node
I20260810 23:38:23.457646 922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:23.458676 922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:23.459964 922 hybrid_clock.cc:648] HybridClock initialized: now 1786405103459870 us; error 48 us; skew 500 ppm
I20260810 23:38:23.460341 922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:23 dist-test-slave-pww0 krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405103, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:23.467351 922 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:23.469086 922 webserver.cc:460] Webserver started at http://127.30.33.2:33729/ using document root <none> and password file <none>
I20260810 23:38:23.469832 922 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:23.469925 922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:23.470212 922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:23.473407 922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "437e2a8836f046ce9844486bbb9b58b2"
format_stamp: "Formatted at 2026-08-10 23:38:23 on dist-test-slave-pww0"
server_key: "9d4370b9f30e5c9d7def30762ea7c263"
server_key_iv: "774d2ca9900c33f86cd348e42dbfda96"
server_key_version: "encryptionkey@0"
I20260810 23:38:23.474213 922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "437e2a8836f046ce9844486bbb9b58b2"
format_stamp: "Formatted at 2026-08-10 23:38:23 on dist-test-slave-pww0"
server_key: "9d4370b9f30e5c9d7def30762ea7c263"
server_key_iv: "774d2ca9900c33f86cd348e42dbfda96"
server_key_version: "encryptionkey@0"
I20260810 23:38:23.480026 922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:38:23.484072 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:23.485620 922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:38:23.485806 922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "437e2a8836f046ce9844486bbb9b58b2"
format_stamp: "Formatted at 2026-08-10 23:38:23 on dist-test-slave-pww0"
server_key: "9d4370b9f30e5c9d7def30762ea7c263"
server_key_iv: "774d2ca9900c33f86cd348e42dbfda96"
server_key_version: "encryptionkey@0"
I20260810 23:38:23.486037 922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:23.517464 922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:23.522529 922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:23.522836 922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:23.526155 922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:23.526275 922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:23.526436 922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:23.526510 922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:38:23 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405103, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:23.556180 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:23.527776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35735 (local address 127.30.33.2:44095)
0810 23:38:23.528713 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:23.528780 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:23.531595 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:23.533037 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:23.533054 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:23.534132 (+ 1078us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:23.535373 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:23.535415 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.536939 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.536945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:23.539813 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:23.539840 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.541040 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.541055 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:23.541456 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:23.542311 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:23.542343 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:23.546062 (+ 3719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:23.550551 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:23.550568 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:23.550593 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:23.551142 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:23.551681 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:23.551686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:23.551689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:23.551995 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:23.552699 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:23.552712 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:23.553319 (+ 607us) client_negotiation.cc:769] Sending connection context
0810 23:38:23.553719 (+ 400us) client_negotiation.cc:240] Negotiation successful
0810 23:38:23.554375 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":363,"threads_started":1}
I20260810 23:38:23.556391 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:23.527042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:44095 (local address 127.30.33.62:35735)
0810 23:38:23.527533 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:23.527539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:23.528938 (+ 1399us) server_negotiation.cc:407] Connection header received
0810 23:38:23.532095 (+ 3157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:23.532103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:23.532218 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:23.532328 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:23.535698 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.536703 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.540036 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.540696 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.540851 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:23.546422 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:23.546431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:23.546436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:23.546487 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:23.550318 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:23.551326 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:23.551333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:23.551337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:23.551555 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:23.552142 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:23.552146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:23.552149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:23.552539 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:23.552730 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:23.555725 (+ 2995us) server_negotiation.cc:299] Negotiation successful
0810 23:38:23.556019 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260810 23:38:23.578071 922 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:35887
I20260810 23:38:23.578090 1196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:35887 every 8 connection(s)
I20260810 23:38:23.579746 922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:23.581594 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:23.566071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35735 (local address 127.30.33.2:60353)
0810 23:38:23.566952 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:23.566977 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:23.567171 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:23.567777 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:23.567782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:23.568447 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:23.569076 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:23.569098 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.570444 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.570449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:23.571607 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:23.571626 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.572755 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.572760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:23.572900 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:23.573524 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:23.573543 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:23.574389 (+ 846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:23.578407 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:23.578412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:23.578414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:23.578822 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:23.579277 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:23.579281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:23.579284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:23.579445 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:23.580261 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:23.580267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:23.580531 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 23:38:23.580732 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 23:38:23.581161 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":484,"threads_started":1}
I20260810 23:38:23.581568 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 922
I20260810 23:38:23.581797 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:23.582361 30852 external_mini_cluster.cc:1428] Setting key b7695a93d92476b757c51a5c048de849
I20260810 23:38:23.582521 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:23.565958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:60353 (local address 127.30.33.62:35735)
0810 23:38:23.566150 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:23.566156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:23.567094 (+ 938us) server_negotiation.cc:407] Connection header received
0810 23:38:23.567341 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:23.567345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:23.567418 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:23.567556 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:23.569258 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.570294 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.571788 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:23.572544 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:23.572699 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:23.574611 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:23.574619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:23.574625 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:23.574670 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:23.578269 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:23.578951 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:23.578957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:23.578961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:23.579127 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:23.579619 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:23.579624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:23.579627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:23.580100 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:23.580284 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:23.581954 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0810 23:38:23.582207 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:38:23.584021 1197 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35735
I20260810 23:38:23.584307 1197 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:23.585049 1197 heartbeater.cc:511] Master 127.30.33.62:35735 requested a full tablet report, sending...
I20260810 23:38:23.587747 498 ts_manager.cc:194] Registered new tserver with Master: 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2:35887)
I20260810 23:38:23.588650 498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:44095
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:23.630686 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35735
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:23.972494 1205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:23.972837 1205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:23.972986 1205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:23.980510 1205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:23.980675 1205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:23.980806 1205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:38:23.980921 1205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:23.991586 1205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:23.992980 1205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:23.994402 1205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:24.008184 1213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:24.008215 1216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:24.008374 1212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:24.013516 1205 server_base.cc:1034] running on GCE node
I20260810 23:38:24.014264 1205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:24.015286 1205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:24.016551 1205 hybrid_clock.cc:648] HybridClock initialized: now 1786405104016465 us; error 51 us; skew 500 ppm
I20260810 23:38:24.016897 1205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:24.019513 915 heartbeater.cc:503] Master 127.30.33.62:35735 was elected leader, sending a full tablet report...
Aug 10 23:38:24 dist-test-slave-pww0 krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405104, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:24.020216 1205 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:24.022579 1205 webserver.cc:460] Webserver started at http://127.30.33.3:43259/ using document root <none> and password file <none>
I20260810 23:38:24.023384 1205 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:24.023522 1205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:24.023865 1205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:24.026871 1205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de"
format_stamp: "Formatted at 2026-08-10 23:38:24 on dist-test-slave-pww0"
server_key: "fba1493bd3d754f20b5d93d06dcded41"
server_key_iv: "c8d5999b8791919e35eb90012e0bb5c5"
server_key_version: "encryptionkey@0"
I20260810 23:38:24.027762 1205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de"
format_stamp: "Formatted at 2026-08-10 23:38:24 on dist-test-slave-pww0"
server_key: "fba1493bd3d754f20b5d93d06dcded41"
server_key_iv: "c8d5999b8791919e35eb90012e0bb5c5"
server_key_version: "encryptionkey@0"
I20260810 23:38:24.033239 1205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:38:24.036890 1235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:24.038393 1205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:24.038594 1205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de"
format_stamp: "Formatted at 2026-08-10 23:38:24 on dist-test-slave-pww0"
server_key: "fba1493bd3d754f20b5d93d06dcded41"
server_key_iv: "c8d5999b8791919e35eb90012e0bb5c5"
server_key_version: "encryptionkey@0"
I20260810 23:38:24.038825 1205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:24.064618 1205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:24.069269 1205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:24.069561 1205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:24.072480 1205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:24.072585 1205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:24.072724 1205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:24.072789 1205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:38:24 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405104, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:24.099818 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.073163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:40177 (local address 127.30.33.62:35735)
0810 23:38:24.073378 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.073386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.075205 (+ 1819us) server_negotiation.cc:407] Connection header received
0810 23:38:24.078190 (+ 2985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.078197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.078319 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.078401 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:24.081697 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.082909 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.085452 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.086191 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.086339 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.091329 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:24.091335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:24.091339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:24.091381 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:24.095046 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.096376 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.096384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.096388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.096570 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.097343 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.097349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.097352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.097712 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:24.097885 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.099156 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.099414 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:38:24.101789 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.074001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35735 (local address 127.30.33.3:40177)
0810 23:38:24.074969 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.075053 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.077757 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.079084 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.079099 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.080132 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:24.081380 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.081415 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.083066 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.083071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.085199 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.085217 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.086488 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.086499 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.086910 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.087857 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:24.087881 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:24.091021 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:24.095302 (+ 4281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:24.095315 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:24.095336 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:24.096146 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:24.096762 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:24.096768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:24.096772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:24.097173 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:24.097881 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:24.097894 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:24.098562 (+ 668us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.098998 (+ 436us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.099746 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":391,"threads_started":1}
I20260810 23:38:24.118228 1205 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:46517
I20260810 23:38:24.118304 1447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:46517 every 8 connection(s)
I20260810 23:38:24.119846 1205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:24.122129 1450 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35735
I20260810 23:38:24.122526 1450 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:24.123420 1450 heartbeater.cc:511] Master 127.30.33.62:35735 requested a full tablet report, sending...
I20260810 23:38:24.124121 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.109183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:34721 (local address 127.30.33.62:35735)
0810 23:38:24.109393 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.109400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.110176 (+ 776us) server_negotiation.cc:407] Connection header received
0810 23:38:24.110371 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.110378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.110477 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.110610 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:24.111658 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.112726 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.114895 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.115360 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.115521 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.117856 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:24.117861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:24.117865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:24.117903 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:24.121164 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.121971 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.121977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.121980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.122177 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.122666 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.122670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.122672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.123006 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:24.123127 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.123673 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.123877 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:38:24.124403 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.109324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35735 (local address 127.30.33.3:34721)
0810 23:38:24.110049 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.110077 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.110220 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.110649 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.110653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.111139 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:24.111449 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.111491 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.112877 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.112883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.114713 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.114733 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.115607 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.115618 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.115811 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.116575 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:24.116594 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:24.117609 (+ 1015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:24.121317 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:24.121323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:24.121327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:24.121875 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:24.122354 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:24.122360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:24.122363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:24.122528 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:24.123134 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:24.123139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:24.123347 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.123544 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.123955 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":417,"threads_started":1}
I20260810 23:38:24.124958 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 1205
I20260810 23:38:24.125133 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:24.125154 498 ts_manager.cc:194] Registered new tserver with Master: 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3:46517)
I20260810 23:38:24.125557 30852 external_mini_cluster.cc:1428] Setting key d18b6311f9fd7ed82177b9fa47e7c76b
I20260810 23:38:24.126030 498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:40177
I20260810 23:38:24.128544 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:38:24 dist-test-slave-pww0 krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:38:24 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:24.183423 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.169212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.30.33.62:35735)
0810 23:38:24.169447 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.169453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.169653 (+ 200us) server_negotiation.cc:407] Connection header received
0810 23:38:24.169897 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.169902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.169983 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.170082 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:24.171607 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.172397 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.173680 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.174103 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.174204 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.178278 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:24.178284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:24.178288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:24.178333 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:24.180959 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.181689 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.181693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.181696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.181842 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.182205 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.182208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.182210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.182496 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:24.182704 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.182958 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.183146 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:38:24.188166 498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:24.195577 498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:24.233213 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.210770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.30.33.2:35887)
0810 23:38:24.221790 (+ 11020us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.221803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.224148 (+ 2345us) server_negotiation.cc:407] Connection header received
0810 23:38:24.224291 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.224299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.224497 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.224704 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.225481 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.226977 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.230128 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.231639 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.231774 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.232001 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.232704 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.232889 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10823,"thread_start_us":10659,"threads_started":1}
I20260810 23:38:24.233296 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.209446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:46517 (local address 127.0.0.1:58606)
0810 23:38:24.219889 (+ 10443us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.219967 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.220238 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.226013 (+ 5775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.226018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.226044 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.226450 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.226463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.228506 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.228509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.230194 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.230210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.232332 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.232338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.232483 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.232508 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.232610 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.232872 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10069,"mutex_wait_us":55,"thread_start_us":13568,"threads_started":1}
I20260810 23:38:24.233341 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.209756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.30.33.3:46517)
0810 23:38:24.225518 (+ 15762us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.225524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.225558 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:38:24.225636 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.225641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.225828 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.226004 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.226704 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.228352 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.230372 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.231862 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.232004 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.232195 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.232790 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.232970 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15564,"thread_start_us":15344,"threads_started":1}
I20260810 23:38:24.234063 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.210085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:35887 (local address 127.0.0.1:48628)
0810 23:38:24.223525 (+ 13440us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.223549 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.223768 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.224705 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.224710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.224734 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.225215 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.225240 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.227148 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.227154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.229934 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.229950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.232159 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.232166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.232398 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.232461 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.232611 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.233665 (+ 1054us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13253,"mutex_wait_us":912,"thread_start_us":10409,"threads_started":1}
I20260810 23:38:24.234380 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.209657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43097 (local address 127.0.0.1:59438)
0810 23:38:24.222961 (+ 13304us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.222983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.223176 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.224372 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.224381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.224432 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.224930 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.224944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.227030 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.227035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.229934 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.229950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.233161 (+ 3211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.233168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.233631 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.233678 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.233769 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.233910 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13125,"mutex_wait_us":237,"thread_start_us":12915,"threads_started":1}
I20260810 23:38:24.234658 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.210214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.30.33.1:43097)
0810 23:38:24.222295 (+ 12081us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.222304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.223325 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 23:38:24.223441 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.223449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.223842 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.224112 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.225197 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.226825 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.230447 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.232681 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.232806 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.233009 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.234174 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.234363 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11897,"thread_start_us":11733,"threads_started":1}
I20260810 23:38:24.237962 1330 tablet_service.cc:1467] Processing CreateTablet for tablet 667b77270fa844bd943fe62c53ec53bb (DEFAULT_TABLE table=test-table [id=be9105b75f0e4d21a4ff5c5c50d089bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:24.238073 1051 tablet_service.cc:1467] Processing CreateTablet for tablet 667b77270fa844bd943fe62c53ec53bb (DEFAULT_TABLE table=test-table [id=be9105b75f0e4d21a4ff5c5c50d089bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:24.238071 785 tablet_service.cc:1467] Processing CreateTablet for tablet 667b77270fa844bd943fe62c53ec53bb (DEFAULT_TABLE table=test-table [id=be9105b75f0e4d21a4ff5c5c50d089bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:24.239346 1330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667b77270fa844bd943fe62c53ec53bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.239543 785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667b77270fa844bd943fe62c53ec53bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.239559 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667b77270fa844bd943fe62c53ec53bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.252655 1487 tablet_bootstrap.cc:492] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Bootstrap starting.
I20260810 23:38:24.254741 1491 tablet_bootstrap.cc:492] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Bootstrap starting.
I20260810 23:38:24.256247 1489 tablet_bootstrap.cc:492] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Bootstrap starting.
I20260810 23:38:24.259073 1487 tablet_bootstrap.cc:654] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.259912 1489 tablet_bootstrap.cc:654] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.260648 1487 log.cc:824] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:24.261421 1489 log.cc:824] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:24.262490 1491 tablet_bootstrap.cc:654] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.263650 1491 log.cc:824] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:24.264436 1487 tablet_bootstrap.cc:492] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: No bootstrap required, opened a new log
I20260810 23:38:24.264436 1489 tablet_bootstrap.cc:492] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: No bootstrap required, opened a new log
I20260810 23:38:24.264734 1489 ts_tablet_manager.cc:1397] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 23:38:24.264729 1487 ts_tablet_manager.cc:1397] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20260810 23:38:24.272888 1491 tablet_bootstrap.cc:492] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: No bootstrap required, opened a new log
I20260810 23:38:24.273240 1491 ts_tablet_manager.cc:1397] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.000s
I20260810 23:38:24.275650 1487 raft_consensus.cc:348] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.275650 1489 raft_consensus.cc:348] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.276150 1487 raft_consensus.cc:374] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.276149 1489 raft_consensus.cc:374] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.276270 1489 raft_consensus.cc:729] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0996546142b44c76a3016eb5d68abc65, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.276270 1487 raft_consensus.cc:729] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b19f2e3ccc4ab5ae58b48d8c1ff2de, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.276983 1489 consensus_queue.cc:260] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.276983 1487 consensus_queue.cc:260] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.278285 1450 heartbeater.cc:503] Master 127.30.33.62:35735 was elected leader, sending a full tablet report...
I20260810 23:38:24.279047 1487 ts_tablet_manager.cc:1428] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 23:38:24.279424 1489 ts_tablet_manager.cc:1428] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260810 23:38:24.285305 1491 raft_consensus.cc:348] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.285957 1491 raft_consensus.cc:374] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.286055 1491 raft_consensus.cc:729] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e2a8836f046ce9844486bbb9b58b2, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.286742 1491 consensus_queue.cc:260] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.287873 1197 heartbeater.cc:503] Master 127.30.33.62:35735 was elected leader, sending a full tablet report...
I20260810 23:38:24.288419 1491 ts_tablet_manager.cc:1428] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260810 23:38:24.332098 1198 tablet.cc:2369] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:38:24.371729 1451 tablet.cc:2369] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:38:24.426515 1499 raft_consensus.cc:482] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:24.426838 1499 raft_consensus.cc:504] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.428695 1499 leader_election.cc:288] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3:46517), 0996546142b44c76a3016eb5d68abc65 (127.30.33.1:43097)
I20260810 23:38:24.435546 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.429354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:41123 (local address 127.30.33.1:43097)
0810 23:38:24.429683 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.429690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.430123 (+ 433us) server_negotiation.cc:407] Connection header received
0810 23:38:24.430341 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.430345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.430419 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.430499 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.431017 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.431773 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.433795 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.434570 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.434714 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.434828 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.435136 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.435270 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 23:38:24.435583 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.429225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:36753 (local address 127.30.33.3:46517)
0810 23:38:24.429442 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.429448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.430426 (+ 978us) server_negotiation.cc:407] Connection header received
0810 23:38:24.430601 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.430605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.430671 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.430746 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.431333 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.432085 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.433873 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.434662 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.434777 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.434887 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.435184 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.435313 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:38:24.435698 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.429008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43097 (local address 127.30.33.2:41123)
0810 23:38:24.429986 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.430004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.430187 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.430526 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.430531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.430556 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.430871 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.430884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.431906 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.431909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.433588 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.433604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.434827 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.434835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.434976 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.435001 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.435088 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.435240 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":75,"thread_start_us":574,"threads_started":1}
I20260810 23:38:24.436105 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.429168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:46517 (local address 127.30.33.2:36753)
0810 23:38:24.430303 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.430317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.430456 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.430789 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.430793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.430821 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.431184 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.431192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.432238 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.432241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.433715 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.433728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.434887 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.434894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.435025 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.435049 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.435138 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.435270 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":790,"threads_started":1}
I20260810 23:38:24.436878 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667b77270fa844bd943fe62c53ec53bb" candidate_uuid: "437e2a8836f046ce9844486bbb9b58b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0996546142b44c76a3016eb5d68abc65" is_pre_election: true
I20260810 23:38:24.437137 1357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667b77270fa844bd943fe62c53ec53bb" candidate_uuid: "437e2a8836f046ce9844486bbb9b58b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" is_pre_election: true
I20260810 23:38:24.437378 823 raft_consensus.cc:2454] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437e2a8836f046ce9844486bbb9b58b2 in term 0.
I20260810 23:38:24.437676 1357 raft_consensus.cc:2454] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437e2a8836f046ce9844486bbb9b58b2 in term 0.
I20260810 23:38:24.438135 948 leader_election.cc:302] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0996546142b44c76a3016eb5d68abc65, 437e2a8836f046ce9844486bbb9b58b2; no voters:
I20260810 23:38:24.438557 1499 raft_consensus.cc:2781] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:24.438716 1499 raft_consensus.cc:482] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:24.438822 1499 raft_consensus.cc:3037] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.441103 1499 raft_consensus.cc:504] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.441761 1499 leader_election.cc:288] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [CANDIDATE]: Term 1 election: Requested vote from peers 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3:46517), 0996546142b44c76a3016eb5d68abc65 (127.30.33.1:43097)
I20260810 23:38:24.442286 1357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667b77270fa844bd943fe62c53ec53bb" candidate_uuid: "437e2a8836f046ce9844486bbb9b58b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de"
I20260810 23:38:24.442338 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667b77270fa844bd943fe62c53ec53bb" candidate_uuid: "437e2a8836f046ce9844486bbb9b58b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0996546142b44c76a3016eb5d68abc65"
I20260810 23:38:24.442498 1357 raft_consensus.cc:3037] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.442566 823 raft_consensus.cc:3037] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.444957 1357 raft_consensus.cc:2454] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437e2a8836f046ce9844486bbb9b58b2 in term 1.
I20260810 23:38:24.445070 823 raft_consensus.cc:2454] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437e2a8836f046ce9844486bbb9b58b2 in term 1.
I20260810 23:38:24.445494 951 leader_election.cc:302] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e2a8836f046ce9844486bbb9b58b2, 69b19f2e3ccc4ab5ae58b48d8c1ff2de; no voters:
I20260810 23:38:24.445859 1499 raft_consensus.cc:2781] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:24.446307 1499 raft_consensus.cc:686] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 1 LEADER]: Becoming Leader. State: Replica: 437e2a8836f046ce9844486bbb9b58b2, State: Running, Role: LEADER
I20260810 23:38:24.446818 1499 consensus_queue.cc:237] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } }
I20260810 23:38:24.452071 496 catalog_manager.cc:5495] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2). New cstate: current_term: 1 leader_uuid: "437e2a8836f046ce9844486bbb9b58b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } health_report { overall_health: HEALTHY } } }
I20260810 23:38:24.483065 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.477587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.30.33.2:35887)
0810 23:38:24.477805 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.477812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.477991 (+ 179us) server_negotiation.cc:407] Connection header received
0810 23:38:24.478287 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.478290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.478388 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.478511 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:24.478827 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.479604 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.480968 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.481389 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.481485 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.482008 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:24.482564 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:24.482671 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.482776 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.482849 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:38:24.491938 823 raft_consensus.cc:1261] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Refusing update from remote peer 437e2a8836f046ce9844486bbb9b58b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:24.491933 1357 raft_consensus.cc:1261] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Refusing update from remote peer 437e2a8836f046ce9844486bbb9b58b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:24.492884 1503 consensus_queue.cc:1035] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:24.493273 1499 consensus_queue.cc:1035] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:24.530416 498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:24.530776 498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:24.534914 498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8: Sending DeleteTablet for 3 replicas of tablet 667b77270fa844bd943fe62c53ec53bb
I20260810 23:38:24.536576 1330 tablet_service.cc:1514] Processing DeleteTablet for tablet 667b77270fa844bd943fe62c53ec53bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:24 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:58606
I20260810 23:38:24.536587 785 tablet_service.cc:1514] Processing DeleteTablet for tablet 667b77270fa844bd943fe62c53ec53bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:24 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:59438
I20260810 23:38:24.536902 1051 tablet_service.cc:1514] Processing DeleteTablet for tablet 667b77270fa844bd943fe62c53ec53bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:24 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:48628
I20260810 23:38:24.542972 498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:24.543851 498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:24.550789 1535 tablet_replica.cc:330] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: stopping tablet replica
I20260810 23:38:24.551061 1531 tablet_replica.cc:330] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: stopping tablet replica
I20260810 23:38:24.551559 1535 raft_consensus.cc:2229] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:24.551687 1531 raft_consensus.cc:2229] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:24.552160 1535 raft_consensus.cc:2258] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:24.552924 1531 raft_consensus.cc:2258] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:24.554549 1329 tablet_service.cc:1467] Processing CreateTablet for tablet 471d6ce1c51f47359e1b07187abd9a2e (DEFAULT_TABLE table=test-table [id=fd3b12a3a18648c48ad380ac0eb3c617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:24.554672 1524 tablet_replica.cc:330] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: stopping tablet replica
I20260810 23:38:24.554760 785 tablet_service.cc:1467] Processing CreateTablet for tablet 471d6ce1c51f47359e1b07187abd9a2e (DEFAULT_TABLE table=test-table [id=fd3b12a3a18648c48ad380ac0eb3c617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:24.555131 1329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471d6ce1c51f47359e1b07187abd9a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.555202 785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471d6ce1c51f47359e1b07187abd9a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.555169 1524 raft_consensus.cc:2229] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:24.555758 1524 raft_consensus.cc:2258] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:24.556334 1051 tablet_service.cc:1467] Processing CreateTablet for tablet 471d6ce1c51f47359e1b07187abd9a2e (DEFAULT_TABLE table=test-table [id=fd3b12a3a18648c48ad380ac0eb3c617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:24.556835 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471d6ce1c51f47359e1b07187abd9a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.558707 1535 ts_tablet_manager.cc:1905] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:24.562302 1487 tablet_bootstrap.cc:492] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Bootstrap starting.
I20260810 23:38:24.564057 1487 tablet_bootstrap.cc:654] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.565176 1531 ts_tablet_manager.cc:1905] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:24.567831 1524 ts_tablet_manager.cc:1905] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:24.568773 1491 tablet_bootstrap.cc:492] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: Bootstrap starting.
I20260810 23:38:24.570163 1535 ts_tablet_manager.cc:1918] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:24.570701 1491 tablet_bootstrap.cc:654] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.577654 1535 log.cc:1196] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/667b77270fa844bd943fe62c53ec53bb
I20260810 23:38:24.578415 1535 ts_tablet_manager.cc:1939] T 667b77270fa844bd943fe62c53ec53bb P 0996546142b44c76a3016eb5d68abc65: Deleting consensus metadata
I20260810 23:38:24.580785 470 catalog_manager.cc:4841] TS 0996546142b44c76a3016eb5d68abc65 (127.30.33.1:43097): tablet 667b77270fa844bd943fe62c53ec53bb (table test-table [id=be9105b75f0e4d21a4ff5c5c50d089bb]) successfully deleted
I20260810 23:38:24.577316 1524 ts_tablet_manager.cc:1918] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:24.581436 1524 log.cc:1196] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/667b77270fa844bd943fe62c53ec53bb
I20260810 23:38:24.582056 1524 ts_tablet_manager.cc:1939] T 667b77270fa844bd943fe62c53ec53bb P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Deleting consensus metadata
I20260810 23:38:24.584052 476 catalog_manager.cc:4841] TS 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3:46517): tablet 667b77270fa844bd943fe62c53ec53bb (table test-table [id=be9105b75f0e4d21a4ff5c5c50d089bb]) successfully deleted
I20260810 23:38:24.584626 1531 ts_tablet_manager.cc:1918] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:24.584851 1531 log.cc:1196] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/667b77270fa844bd943fe62c53ec53bb
I20260810 23:38:24.585546 1531 ts_tablet_manager.cc:1939] T 667b77270fa844bd943fe62c53ec53bb P 437e2a8836f046ce9844486bbb9b58b2: Deleting consensus metadata
I20260810 23:38:24.587567 474 catalog_manager.cc:4841] TS 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2:35887): tablet 667b77270fa844bd943fe62c53ec53bb (table test-table [id=be9105b75f0e4d21a4ff5c5c50d089bb]) successfully deleted
I20260810 23:38:24.591544 1491 tablet_bootstrap.cc:492] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: No bootstrap required, opened a new log
I20260810 23:38:24.591742 1491 ts_tablet_manager.cc:1397] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20260810 23:38:24.592305 1489 tablet_bootstrap.cc:492] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Bootstrap starting.
I20260810 23:38:24.592672 1491 raft_consensus.cc:348] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.592914 1491 raft_consensus.cc:374] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.593029 1491 raft_consensus.cc:729] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e2a8836f046ce9844486bbb9b58b2, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.594010 1489 tablet_bootstrap.cc:654] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.594501 1487 tablet_bootstrap.cc:492] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: No bootstrap required, opened a new log
I20260810 23:38:24.594681 1487 ts_tablet_manager.cc:1397] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.011s
I20260810 23:38:24.595606 1487 raft_consensus.cc:348] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.595866 1487 raft_consensus.cc:374] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.595965 1487 raft_consensus.cc:729] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b19f2e3ccc4ab5ae58b48d8c1ff2de, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.596388 1491 consensus_queue.cc:260] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.597100 1491 ts_tablet_manager.cc:1428] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 23:38:24.597487 1487 consensus_queue.cc:260] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.599112 1487 ts_tablet_manager.cc:1428] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:38:24.605213 1489 tablet_bootstrap.cc:492] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: No bootstrap required, opened a new log
I20260810 23:38:24.605392 1489 ts_tablet_manager.cc:1397] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260810 23:38:24.606073 1489 raft_consensus.cc:348] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.606269 1489 raft_consensus.cc:374] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.606366 1489 raft_consensus.cc:729] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0996546142b44c76a3016eb5d68abc65, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.606684 1489 consensus_queue.cc:260] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.608273 1489 ts_tablet_manager.cc:1428] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 23:38:24.610011 1498 raft_consensus.cc:482] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:24.610206 1498 raft_consensus.cc:504] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.612061 1498 leader_election.cc:288] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2:35887), 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3:46517)
I20260810 23:38:24.620520 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.613262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:42629 (local address 127.30.33.2:35887)
0810 23:38:24.613587 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.613592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.613610 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:38:24.613687 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.613692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.613772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.613862 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.614788 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.616030 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.618185 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.619202 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.619304 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.619381 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.620030 (+ 649us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.620186 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260810 23:38:24.620534 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.612298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:35887 (local address 127.30.33.1:42629)
0810 23:38:24.612862 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.612883 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.613007 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.614087 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.614093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.614125 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.614609 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.614621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.616149 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.616154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.617995 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.618012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.619608 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.619615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.619836 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.619873 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.619975 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.620122 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":1,"thread_start_us":1410,"threads_started":1}
I20260810 23:38:24.621362 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471d6ce1c51f47359e1b07187abd9a2e" candidate_uuid: "0996546142b44c76a3016eb5d68abc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e2a8836f046ce9844486bbb9b58b2" is_pre_election: true
I20260810 23:38:24.621763 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.613249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:53681 (local address 127.30.33.3:46517)
0810 23:38:24.613428 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.613434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.614859 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 23:38:24.615011 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.615017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.615113 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.615204 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.616028 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.617230 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.619650 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.620809 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.620926 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.621017 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.621370 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.621498 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:38:24.621763 1540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.613442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:46517 (local address 127.30.33.1:53681)
0810 23:38:24.614738 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.614759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.614897 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.615386 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.615392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.615418 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.615810 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.615823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.617402 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.617408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.619442 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.619495 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.621004 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.621010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.621201 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.621241 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.621321 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.621445 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"thread_start_us":889,"threads_started":1}
I20260810 23:38:24.621896 1094 raft_consensus.cc:2454] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0996546142b44c76a3016eb5d68abc65 in term 0.
I20260810 23:38:24.622495 1357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471d6ce1c51f47359e1b07187abd9a2e" candidate_uuid: "0996546142b44c76a3016eb5d68abc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" is_pre_election: true
I20260810 23:38:24.622757 1357 raft_consensus.cc:2454] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0996546142b44c76a3016eb5d68abc65 in term 0.
I20260810 23:38:24.622900 689 leader_election.cc:302] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0996546142b44c76a3016eb5d68abc65, 437e2a8836f046ce9844486bbb9b58b2; no voters:
I20260810 23:38:24.623353 1498 raft_consensus.cc:2781] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:24.623525 1498 raft_consensus.cc:482] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:24.623628 1498 raft_consensus.cc:3037] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.625964 1498 raft_consensus.cc:504] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.626688 1498 leader_election.cc:288] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [CANDIDATE]: Term 1 election: Requested vote from peers 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2:35887), 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3:46517)
I20260810 23:38:24.627277 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471d6ce1c51f47359e1b07187abd9a2e" candidate_uuid: "0996546142b44c76a3016eb5d68abc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e2a8836f046ce9844486bbb9b58b2"
I20260810 23:38:24.627324 1357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471d6ce1c51f47359e1b07187abd9a2e" candidate_uuid: "0996546142b44c76a3016eb5d68abc65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de"
I20260810 23:38:24.627573 1094 raft_consensus.cc:3037] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.627640 1357 raft_consensus.cc:3037] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.630667 1357 raft_consensus.cc:2454] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0996546142b44c76a3016eb5d68abc65 in term 1.
I20260810 23:38:24.630966 1094 raft_consensus.cc:2454] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0996546142b44c76a3016eb5d68abc65 in term 1.
I20260810 23:38:24.631600 689 leader_election.cc:302] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0996546142b44c76a3016eb5d68abc65, 437e2a8836f046ce9844486bbb9b58b2, 69b19f2e3ccc4ab5ae58b48d8c1ff2de; no voters:
I20260810 23:38:24.632083 1498 raft_consensus.cc:2781] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:24.632622 1498 raft_consensus.cc:686] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 1 LEADER]: Becoming Leader. State: Replica: 0996546142b44c76a3016eb5d68abc65, State: Running, Role: LEADER
I20260810 23:38:24.633054 1498 consensus_queue.cc:237] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.636690 496 catalog_manager.cc:5495] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0996546142b44c76a3016eb5d68abc65 (127.30.33.1). New cstate: current_term: 1 leader_uuid: "0996546142b44c76a3016eb5d68abc65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:38:24 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405102, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:24.698873 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.682228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35735 (local address 127.0.0.1:45640)
0810 23:38:24.683013 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.683042 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.683192 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.683593 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.683597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.684405 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:24.684688 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.684698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.686289 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.686293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.687491 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.687511 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.688650 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.688655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.688796 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.689541 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:24.689567 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:24.692301 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:24.696224 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:24.696240 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:24.696246 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:24.696715 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:24.697148 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:24.697151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:24.697153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:24.697320 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:24.697951 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:24.697958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:24.698142 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.698284 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.698522 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":413,"threads_started":1}
I20260810 23:38:24.699105 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.682242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.30.33.62:35735)
0810 23:38:24.682501 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.682508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.683141 (+ 633us) server_negotiation.cc:407] Connection header received
0810 23:38:24.683309 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.683315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.683404 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.683566 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:24.684852 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.686117 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.687691 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.688446 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.688583 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.692517 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:24.692525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:24.692531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:24.692576 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:24.696043 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.696863 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.696870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.696874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.697045 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:24.697429 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:24.697435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:24.697438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:24.697822 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:24.698008 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.698434 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.698697 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:38:24.704814 496 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:45640:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:38:24.706054 496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:24.713289 785 tablet_service.cc:1467] Processing CreateTablet for tablet d08940c275ef4322a784bebc109fd780 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=deadc1143b7246feb474a2ea68fc98cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:38:24.713984 785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08940c275ef4322a784bebc109fd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.713766 1051 tablet_service.cc:1467] Processing CreateTablet for tablet d08940c275ef4322a784bebc109fd780 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=deadc1143b7246feb474a2ea68fc98cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:38:24.714000 1329 tablet_service.cc:1467] Processing CreateTablet for tablet d08940c275ef4322a784bebc109fd780 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=deadc1143b7246feb474a2ea68fc98cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:38:24.714344 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08940c275ef4322a784bebc109fd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.714551 1329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08940c275ef4322a784bebc109fd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:24.724411 1491 tablet_bootstrap.cc:492] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: Bootstrap starting.
I20260810 23:38:24.724489 1489 tablet_bootstrap.cc:492] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: Bootstrap starting.
I20260810 23:38:24.725895 1487 tablet_bootstrap.cc:492] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Bootstrap starting.
I20260810 23:38:24.726341 1489 tablet_bootstrap.cc:654] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.726374 1491 tablet_bootstrap.cc:654] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.727663 1487 tablet_bootstrap.cc:654] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:24.737596 1491 tablet_bootstrap.cc:492] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: No bootstrap required, opened a new log
I20260810 23:38:24.737610 1489 tablet_bootstrap.cc:492] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: No bootstrap required, opened a new log
I20260810 23:38:24.737810 1489 ts_tablet_manager.cc:1397] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260810 23:38:24.737972 1491 ts_tablet_manager.cc:1397] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260810 23:38:24.738718 1489 raft_consensus.cc:348] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.739019 1489 raft_consensus.cc:374] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.739102 1489 raft_consensus.cc:729] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0996546142b44c76a3016eb5d68abc65, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.738864 1491 raft_consensus.cc:348] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.739367 1491 raft_consensus.cc:374] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.739517 1491 raft_consensus.cc:729] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e2a8836f046ce9844486bbb9b58b2, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.739439 1489 consensus_queue.cc:260] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.740010 1498 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:24.740351 1498 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:24.740859 1489 ts_tablet_manager.cc:1428] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 23:38:24.742264 1491 consensus_queue.cc:260] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.742970 1515 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:24.743376 1515 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:24.743685 1487 tablet_bootstrap.cc:492] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: No bootstrap required, opened a new log
I20260810 23:38:24.743881 1487 ts_tablet_manager.cc:1397] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20260810 23:38:24.744712 1487 raft_consensus.cc:348] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.745152 1487 raft_consensus.cc:374] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:24.745245 1487 raft_consensus.cc:729] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b19f2e3ccc4ab5ae58b48d8c1ff2de, State: Initialized, Role: FOLLOWER
I20260810 23:38:24.745535 1487 consensus_queue.cc:260] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.746218 1497 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:24.746457 1491 ts_tablet_manager.cc:1428] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20260810 23:38:24.746556 1497 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:24.746989 1487 ts_tablet_manager.cc:1428] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260810 23:38:24.763324 919 tablet.cc:2369] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:38:24.973534 1497 raft_consensus.cc:482] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:24.973798 1497 raft_consensus.cc:504] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.975370 1497 leader_election.cc:288] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0996546142b44c76a3016eb5d68abc65 (127.30.33.1:43097), 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2:35887)
I20260810 23:38:24.982571 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.976569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:53191 (local address 127.30.33.2:35887)
0810 23:38:24.976756 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.976763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.976785 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:38:24.976889 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.976894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.976976 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.977082 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.977625 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.978773 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.980696 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.981570 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.981710 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.981857 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.982127 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.982281 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:38:24.982589 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.975746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:35887 (local address 127.30.33.3:53191)
0810 23:38:24.976494 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.976510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.976604 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.977126 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.977132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.977161 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.977449 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.977464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.978927 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.978934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.980516 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.980530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.981740 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.981746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.981938 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.981984 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.982082 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.982236 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":907,"threads_started":1}
I20260810 23:38:24.983973 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.975606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43097 (local address 127.30.33.3:45411)
0810 23:38:24.976227 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:24.976248 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:24.976407 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:24.977165 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:24.977169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:24.977186 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:24.977979 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.977988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.979353 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.979363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.981474 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:24.981493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.983121 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.983128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:24.983319 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.983357 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:38:24.983538 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 23:38:24.983675 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":310,"threads_started":1}
I20260810 23:38:24.983989 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:24.976264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:45411 (local address 127.30.33.1:43097)
0810 23:38:24.976878 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:24.976883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:24.976899 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:38:24.976960 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:24.976963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:24.977031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:24.978119 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:24.978257 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.979209 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.981671 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:24.982848 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:24.983021 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:24.983134 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:24.983572 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 23:38:24.983695 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":950,"threads_started":1}
I20260810 23:38:24.984226 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08940c275ef4322a784bebc109fd780" candidate_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e2a8836f046ce9844486bbb9b58b2" is_pre_election: true
I20260810 23:38:24.984535 1094 raft_consensus.cc:2454] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69b19f2e3ccc4ab5ae58b48d8c1ff2de in term 0.
I20260810 23:38:24.984807 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08940c275ef4322a784bebc109fd780" candidate_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0996546142b44c76a3016eb5d68abc65" is_pre_election: true
I20260810 23:38:24.985047 823 raft_consensus.cc:2454] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69b19f2e3ccc4ab5ae58b48d8c1ff2de in term 0.
I20260810 23:38:24.985266 1246 leader_election.cc:302] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e2a8836f046ce9844486bbb9b58b2, 69b19f2e3ccc4ab5ae58b48d8c1ff2de; no voters:
I20260810 23:38:24.985746 1497 raft_consensus.cc:2781] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:24.985885 1497 raft_consensus.cc:482] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:24.985991 1497 raft_consensus.cc:3037] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.988544 1497 raft_consensus.cc:504] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.989243 1497 leader_election.cc:288] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [CANDIDATE]: Term 1 election: Requested vote from peers 0996546142b44c76a3016eb5d68abc65 (127.30.33.1:43097), 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2:35887)
I20260810 23:38:24.989876 823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08940c275ef4322a784bebc109fd780" candidate_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0996546142b44c76a3016eb5d68abc65"
I20260810 23:38:24.990110 823 raft_consensus.cc:3037] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.991799 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08940c275ef4322a784bebc109fd780" candidate_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e2a8836f046ce9844486bbb9b58b2"
I20260810 23:38:24.992050 1094 raft_consensus.cc:3037] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:24.993166 823 raft_consensus.cc:2454] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69b19f2e3ccc4ab5ae58b48d8c1ff2de in term 1.
I20260810 23:38:24.993683 1237 leader_election.cc:302] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0996546142b44c76a3016eb5d68abc65, 69b19f2e3ccc4ab5ae58b48d8c1ff2de; no voters:
I20260810 23:38:24.994000 1497 raft_consensus.cc:2781] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:24.994463 1497 raft_consensus.cc:686] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 LEADER]: Becoming Leader. State: Replica: 69b19f2e3ccc4ab5ae58b48d8c1ff2de, State: Running, Role: LEADER
I20260810 23:38:24.994735 1094 raft_consensus.cc:2454] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69b19f2e3ccc4ab5ae58b48d8c1ff2de in term 1.
I20260810 23:38:24.994907 1497 consensus_queue.cc:237] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } }
I20260810 23:38:24.997382 1571 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: TxnStatusTablet state changed. Reason: New leader 69b19f2e3ccc4ab5ae58b48d8c1ff2de. Latest consensus state: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:24.997767 1571 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:38:24.998670 498 catalog_manager.cc:5495] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de reported cstate change: term changed from 0 to 1, leader changed from <none> to 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3). New cstate: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } health_report { overall_health: HEALTHY } } }
I20260810 23:38:24.999245 1575 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:38:24.999553 1575 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:38:25.030177 1498 consensus_queue.cc:1035] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:25.041321 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.032676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:46517 (local address 127.0.0.1:58608)
0810 23:38:25.033201 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:25.033233 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:25.033406 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:25.034017 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:25.034023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:25.034037 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:38:25.034266 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:25.034280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.035683 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.035687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:25.037232 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:25.037252 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.038557 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.038564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:25.038810 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.038912 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:38:25.040459 (+ 1547us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:38:25.040493 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:38:25.040578 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:38:25.040948 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260810 23:38:25.042135 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.033080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.30.33.3:46517)
0810 23:38:25.033421 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.033427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.033448 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:38:25.033700 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.033706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.033817 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.033967 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:25.034432 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.035554 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.037443 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.038253 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.038378 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.039192 (+ 814us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:25.040199 (+ 1007us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:25.040386 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.041653 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.041785 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260810 23:38:25.051231 1550 consensus_queue.cc:1035] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:25.053721 1090 raft_consensus.cc:1261] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Refusing update from remote peer 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:25.054612 823 raft_consensus.cc:1261] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Refusing update from remote peer 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:25.056914 1571 consensus_queue.cc:1035] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:25.057757 1571 consensus_queue.cc:1035] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:25.062952 1550 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: TxnStatusTablet state changed. Reason: New leader 69b19f2e3ccc4ab5ae58b48d8c1ff2de. Latest consensus state: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:25.063375 1550 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:25.066141 1583 mvcc.cc:205] Tried to move back new op lower bound from 7317115310288220160 to 7317115310063624192. Current Snapshot: MvccSnapshot[applied={T|T < 7317115310288220160}]
I20260810 23:38:25.066221 1515 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: TxnStatusTablet state changed. Reason: New leader 69b19f2e3ccc4ab5ae58b48d8c1ff2de. Latest consensus state: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:25.066551 1515 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:25.067540 1497 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:25.067883 1497 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:38:25.072608 1571 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:25.072872 1571 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:38:25.073707 1498 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:25.073992 1498 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 0996546142b44c76a3016eb5d68abc65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:25.074549 1580 mvcc.cc:205] Tried to move back new op lower bound from 7317115310288220160 to 7317115310063624192. Current Snapshot: MvccSnapshot[applied={T|T < 7317115310288220160}]
I20260810 23:38:25.078789 1586 mvcc.cc:205] Tried to move back new op lower bound from 7317115310288220160 to 7317115310063624192. Current Snapshot: MvccSnapshot[applied={T|T < 7317115310288220160}]
I20260810 23:38:25.084656 1515 tablet_replica.cc:439] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0996546142b44c76a3016eb5d68abc65" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43097 } } peers { permanent_uuid: "437e2a8836f046ce9844486bbb9b58b2" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 35887 } } peers { permanent_uuid: "69b19f2e3ccc4ab5ae58b48d8c1ff2de" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 46517 } } }
I20260810 23:38:25.084939 1515 tablet_replica.cc:442] T d08940c275ef4322a784bebc109fd780 P 437e2a8836f046ce9844486bbb9b58b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:25.092491 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.083790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.30.33.1:43097)
0810 23:38:25.084185 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.084192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.084792 (+ 600us) server_negotiation.cc:407] Connection header received
0810 23:38:25.085140 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.085147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.085280 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.085449 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:25.086200 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.087325 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.089896 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.090472 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.090594 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.091215 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:25.091861 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:25.092018 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.092126 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.092242 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260810 23:38:25.109994 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.103569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:46517 (local address 127.30.33.1:33441)
0810 23:38:25.104330 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:25.104347 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:25.104499 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:25.104870 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:25.104874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:25.104886 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:38:25.105178 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:25.105188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.106667 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.106671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:25.107774 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:25.107788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.108883 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.108887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:25.109006 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.109069 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:38:25.109505 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:38:25.109514 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:38:25.109569 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 23:38:25.109687 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":395,"threads_started":1}
I20260810 23:38:25.110049 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.103627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:33441 (local address 127.30.33.3:46517)
0810 23:38:25.103843 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.103850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.104436 (+ 586us) server_negotiation.cc:407] Connection header received
0810 23:38:25.104627 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.104640 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.104739 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.104851 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:25.105335 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.106518 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.107911 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.108698 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.108848 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.109251 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:25.109418 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:25.109510 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.109606 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.109706 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:38:25.175827 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.168964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:59511 (local address 127.30.33.1:43097)
0810 23:38:25.169148 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.169155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.169782 (+ 627us) server_negotiation.cc:407] Connection header received
0810 23:38:25.169979 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.169984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.170076 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.170169 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:25.170919 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.171875 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.173739 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.174439 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.174576 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.175020 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:25.175177 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:25.175264 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.175398 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.175526 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:38:25.175886 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.168840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43097 (local address 127.30.33.3:59511)
0810 23:38:25.169657 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:25.169676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:25.169833 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:25.170222 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:25.170229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:25.170258 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:38:25.170681 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:25.170694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.172056 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.172062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:25.173517 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:25.173532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.174643 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.174649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:25.174781 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.174845 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:38:25.175289 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:38:25.175299 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:38:25.175358 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:38:25.175506 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":587,"threads_started":1}
I20260810 23:38:25.225489 500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:25.225785 500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:25.229597 500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1faef59025114e05b5044ec76f067da8: Sending DeleteTablet for 3 replicas of tablet 471d6ce1c51f47359e1b07187abd9a2e
I20260810 23:38:25.230621 785 tablet_service.cc:1514] Processing DeleteTablet for tablet 471d6ce1c51f47359e1b07187abd9a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:25 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:59438
I20260810 23:38:25.230799 1051 tablet_service.cc:1514] Processing DeleteTablet for tablet 471d6ce1c51f47359e1b07187abd9a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:25 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:48628
I20260810 23:38:25.231760 1329 tablet_service.cc:1514] Processing DeleteTablet for tablet 471d6ce1c51f47359e1b07187abd9a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:25 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:58606
I20260810 23:38:25.239967 1634 tablet_replica.cc:330] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: stopping tablet replica
I20260810 23:38:25.240269 1634 raft_consensus.cc:2229] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:25.240612 1634 raft_consensus.cc:2258] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:25.242019 1634 ts_tablet_manager.cc:1905] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:25.248178 1630 tablet_replica.cc:330] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: stopping tablet replica
I20260810 23:38:25.248426 1630 raft_consensus.cc:2229] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:25.248670 1630 raft_consensus.cc:2258] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:25.249910 1630 ts_tablet_manager.cc:1905] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:25.255649 1629 tablet_replica.cc:330] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: stopping tablet replica
I20260810 23:38:25.256705 1634 ts_tablet_manager.cc:1918] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:25.256862 1634 log.cc:1196] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/471d6ce1c51f47359e1b07187abd9a2e
I20260810 23:38:25.256722 1629 raft_consensus.cc:2229] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:25.257321 1634 ts_tablet_manager.cc:1939] T 471d6ce1c51f47359e1b07187abd9a2e P 69b19f2e3ccc4ab5ae58b48d8c1ff2de: Deleting consensus metadata
I20260810 23:38:25.257627 1630 ts_tablet_manager.cc:1918] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:25.257782 1630 log.cc:1196] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/471d6ce1c51f47359e1b07187abd9a2e
I20260810 23:38:25.258208 1630 ts_tablet_manager.cc:1939] T 471d6ce1c51f47359e1b07187abd9a2e P 437e2a8836f046ce9844486bbb9b58b2: Deleting consensus metadata
I20260810 23:38:25.259231 1629 raft_consensus.cc:2258] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:25.259343 474 catalog_manager.cc:4841] TS 437e2a8836f046ce9844486bbb9b58b2 (127.30.33.2:35887): tablet 471d6ce1c51f47359e1b07187abd9a2e (table test-table [id=fd3b12a3a18648c48ad380ac0eb3c617]) successfully deleted
I20260810 23:38:25.260577 1629 ts_tablet_manager.cc:1905] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:25.261610 476 catalog_manager.cc:4841] TS 69b19f2e3ccc4ab5ae58b48d8c1ff2de (127.30.33.3:46517): tablet 471d6ce1c51f47359e1b07187abd9a2e (table test-table [id=fd3b12a3a18648c48ad380ac0eb3c617]) successfully deleted
I20260810 23:38:25.270125 1629 ts_tablet_manager.cc:1918] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:25.270335 1629 log.cc:1196] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/471d6ce1c51f47359e1b07187abd9a2e
I20260810 23:38:25.270867 1629 ts_tablet_manager.cc:1939] T 471d6ce1c51f47359e1b07187abd9a2e P 0996546142b44c76a3016eb5d68abc65: Deleting consensus metadata
I20260810 23:38:25.272267 470 catalog_manager.cc:4841] TS 0996546142b44c76a3016eb5d68abc65 (127.30.33.1:43097): tablet 471d6ce1c51f47359e1b07187abd9a2e (table test-table [id=fd3b12a3a18648c48ad380ac0eb3c617]) successfully deleted
Aug 10 23:38:25 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:25.281853 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.252768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.30.33.1:43097)
0810 23:38:25.252983 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.252991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.256482 (+ 3491us) server_negotiation.cc:407] Connection header received
0810 23:38:25.258101 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.258107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.258208 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.258285 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:25.263661 (+ 5376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.264842 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.267051 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.267885 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.268004 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.273879 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:25.273895 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:25.273916 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:25.273963 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:25.278294 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.279296 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.279308 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.279312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.279527 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.280011 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.280016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.280024 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.280593 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:25.280887 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.281221 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.281472 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 23:38:25.282641 908 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59460
I20260810 23:38:25.317018 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.286355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.30.33.62:35735)
0810 23:38:25.299883 (+ 13528us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.299888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.299911 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:38:25.299978 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.299982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.300051 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.300180 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:25.302201 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.303511 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.305635 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.306334 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.306500 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.309791 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:25.309799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:25.309804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:25.309857 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:25.313642 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.314592 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.314599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.314602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.314787 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.315238 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.315243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.315246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.315680 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:25.315931 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.316356 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.316623 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13381,"thread_start_us":13255,"threads_started":1}
W20260810 23:38:25.317687 500 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656
I20260810 23:38:25.321511 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 649
I20260810 23:38:25.346340 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.324032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.30.33.1:43097)
0810 23:38:25.324263 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.324269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.328045 (+ 3776us) server_negotiation.cc:407] Connection header received
0810 23:38:25.328225 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.328230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.328348 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.328469 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:25.330087 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.331375 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.333199 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.333931 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.334471 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.337294 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:25.337302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:25.337308 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:25.337357 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:25.341551 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.342623 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.342630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.342634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.342899 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.343440 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.343445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.343448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.343997 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:25.344294 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.345609 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.345909 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 23:38:25.347285 908 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59476
W20260810 23:38:25.347998 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 649: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:25.348169 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 649
I20260810 23:38:25.398990 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 922
Aug 10 23:38:25 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:38:25.418061 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.399824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.30.33.2:35887)
0810 23:38:25.399999 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.400004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.400024 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:38:25.400266 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.400271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.400352 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.400429 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:25.402129 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.403178 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.405239 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.405912 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.406049 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.411665 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:25.411683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:25.411700 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:25.411744 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:25.414731 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.415738 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.415747 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.415750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.415921 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.416424 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.416429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.416431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.416834 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:25.417098 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.417542 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.417742 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 23:38:25.418923 1184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48632
W20260810 23:38:25.419736 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 922: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:25.419927 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 922
I20260810 23:38:25.457353 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1205
Aug 10 23:38:25 dist-test-slave-pww0 krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:38:25.475255 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:25.458123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.30.33.3:46517)
0810 23:38:25.458346 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:25.458354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:25.458380 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:38:25.458457 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:25.458462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:25.458551 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:25.458687 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:25.459968 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.461143 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.462579 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:25.463309 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:25.463594 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:25.468419 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:25.468443 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:25.468460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:25.468518 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:25.471961 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.472943 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.472952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.472954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.473132 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:25.473610 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:25.473618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:25.473620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:25.474111 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:25.474428 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:25.474740 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 23:38:25.474926 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 23:38:25.476081 1439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58624
W20260810 23:38:25.476866 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1205: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:25.477021 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 1205
I20260810 23:38:25.523531 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 439
I20260810 23:38:25.524782 583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:25.644425 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 439
2026-08-10T23:38:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5273 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:25 dist-test-slave-pww0 krb5kdc[1654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:25 dist-test-slave-pww0 krb5kdc[1654](info): set up 2 sockets
Aug 10 23:38:25 dist-test-slave-pww0 krb5kdc[1654](info): commencing operation
krb5kdc: starting...
W20260810 23:38:26.922050 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:27 dist-test-slave-pww0 krb5kdc[1654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:27.163525 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:45361
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:27.519389 1674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:27.519835 1674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:27.519985 1674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:27.527698 1674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:27.527851 1674 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:38:27.527931 1674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:27.528033 1674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:27.528115 1674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:27.528213 1674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:27.539002 1674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:45361
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:27.540491 1674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:27.541991 1674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:27.554983 1685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:27.555051 1682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:27.555076 1687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:27.555896 1674 server_base.cc:1034] running on GCE node
I20260810 23:38:27.556763 1674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:27.558374 1674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:27.559619 1674 hybrid_clock.cc:648] HybridClock initialized: now 1786405107559525 us; error 59 us; skew 500 ppm
I20260810 23:38:27.560009 1674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:27 dist-test-slave-pww0 krb5kdc[1654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405107, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:27.563441 1674 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:27.565120 1674 webserver.cc:460] Webserver started at http://127.30.33.62:43911/ using document root <none> and password file <none>
I20260810 23:38:27.566005 1674 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:27.566103 1674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:27.566432 1674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:27.569690 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7"
format_stamp: "Formatted at 2026-08-10 23:38:27 on dist-test-slave-pww0"
server_key: "2c851703630af0aa26e1a48d7c3390b0"
server_key_iv: "1c189220e7835af10d77faaef291af39"
server_key_version: "encryptionkey@0"
I20260810 23:38:27.570585 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7"
format_stamp: "Formatted at 2026-08-10 23:38:27 on dist-test-slave-pww0"
server_key: "2c851703630af0aa26e1a48d7c3390b0"
server_key_iv: "1c189220e7835af10d77faaef291af39"
server_key_version: "encryptionkey@0"
I20260810 23:38:27.576529 1674 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260810 23:38:27.580482 1699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:27.582064 1674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:27.582270 1674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7"
format_stamp: "Formatted at 2026-08-10 23:38:27 on dist-test-slave-pww0"
server_key: "2c851703630af0aa26e1a48d7c3390b0"
server_key_iv: "1c189220e7835af10d77faaef291af39"
server_key_version: "encryptionkey@0"
I20260810 23:38:27.582508 1674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:27.598690 1674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:27.603120 1674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:27.603416 1674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:27.626257 1674 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:45361
I20260810 23:38:27.626271 1806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:45361 every 8 connection(s)
I20260810 23:38:27.627887 1674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:27.631966 1807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:27.637265 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 1674
I20260810 23:38:27.637486 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:27.638166 30852 external_mini_cluster.cc:1428] Setting key 06af3d294920da800ccb8ea75619ba9a
I20260810 23:38:27.643839 1807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Bootstrap starting.
I20260810 23:38:27.648002 1807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:27.649503 1807 log.cc:824] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:27.653403 1807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: No bootstrap required, opened a new log
Aug 10 23:38:27 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:27.661981 1807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45361 } }
I20260810 23:38:27.662438 1807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:27.662570 1807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f9bd8eb6024fe7adec5410e6e8a1f7, State: Initialized, Role: FOLLOWER
I20260810 23:38:27.663252 1807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45361 } }
I20260810 23:38:27.663406 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:27.640393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.30.33.62:45361)
0810 23:38:27.641312 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:27.641325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:27.641378 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:38:27.643233 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:27.643261 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:27.643785 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:27.644191 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:27.646150 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:27.647626 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:27.649489 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:27.650497 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:27.650726 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:27.655359 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:27.655369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:27.655391 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:27.655435 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:27.658795 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:27.659634 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:27.659642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:27.659654 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:27.659851 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:27.660278 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:27.660282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:27.660284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:27.660926 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:27.661231 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:27.661722 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 23:38:27.662181 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":404,"threads_started":1}
I20260810 23:38:27.663558 1807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:27.663702 1807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:27.663895 1807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:27.666285 1807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45361 } }
I20260810 23:38:27.666849 1807 leader_election.cc:302] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55f9bd8eb6024fe7adec5410e6e8a1f7; no voters:
I20260810 23:38:27.667395 1807 leader_election.cc:288] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:27.667557 1817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:27.668219 1817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [term 1 LEADER]: Becoming Leader. State: Replica: 55f9bd8eb6024fe7adec5410e6e8a1f7, State: Running, Role: LEADER
I20260810 23:38:27.668751 1817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45361 } }
I20260810 23:38:27.669466 1807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:27.671572 1818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45361 } } }
I20260810 23:38:27.671655 1822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f9bd8eb6024fe7adec5410e6e8a1f7. Latest consensus state: current_term: 1 leader_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f9bd8eb6024fe7adec5410e6e8a1f7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45361 } } }
I20260810 23:38:27.672029 1818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:27.672041 1822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:27.675254 1827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:27.680817 1827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:27.692150 1827 catalog_manager.cc:1347] Generated new cluster ID: 1e5369ace5244eb9afee4c39697818de
I20260810 23:38:27.692296 1827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:27.724768 1827 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:27.726204 1827 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:27.732025 1827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Generated new TSK 0
I20260810 23:38:27.733004 1827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:27.801323 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45361
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:28.179181 1852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:28.179543 1852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:28.179710 1852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:28.187492 1852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:28.187645 1852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:28.187798 1852 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:38:28.187870 1852 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:38:28.187985 1852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:28.202978 1852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:28.204427 1852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:28.205819 1852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:28.219923 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:28.219981 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:28.220615 1852 server_base.cc:1034] running on GCE node
W20260810 23:38:28.220686 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:28.221410 1852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:28.222409 1852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:28.223603 1852 hybrid_clock.cc:648] HybridClock initialized: now 1786405108223537 us; error 59 us; skew 500 ppm
I20260810 23:38:28.223923 1852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:28 dist-test-slave-pww0 krb5kdc[1654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405108, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:28.227319 1852 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:28.228914 1852 webserver.cc:460] Webserver started at http://127.30.33.1:44015/ using document root <none> and password file <none>
I20260810 23:38:28.229671 1852 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:28.229789 1852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:28.230077 1852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:28.233183 1852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "f66fdbe8480e4284a0f6f53a20232801"
format_stamp: "Formatted at 2026-08-10 23:38:28 on dist-test-slave-pww0"
server_key: "e3f6619a964450739d400f547f913f04"
server_key_iv: "72b3eae6d0e9dcfda065067690218634"
server_key_version: "encryptionkey@0"
I20260810 23:38:28.233906 1852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "f66fdbe8480e4284a0f6f53a20232801"
format_stamp: "Formatted at 2026-08-10 23:38:28 on dist-test-slave-pww0"
server_key: "e3f6619a964450739d400f547f913f04"
server_key_iv: "72b3eae6d0e9dcfda065067690218634"
server_key_version: "encryptionkey@0"
I20260810 23:38:28.239324 1852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:38:28.243027 1884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:28.244645 1852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:28.244833 1852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "f66fdbe8480e4284a0f6f53a20232801"
format_stamp: "Formatted at 2026-08-10 23:38:28 on dist-test-slave-pww0"
server_key: "e3f6619a964450739d400f547f913f04"
server_key_iv: "72b3eae6d0e9dcfda065067690218634"
server_key_version: "encryptionkey@0"
I20260810 23:38:28.245055 1852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:28.267313 1852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:28.272034 1852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:28.272347 1852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:28.275209 1852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:28.275332 1852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:28.275475 1852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:28.275563 1852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 10 23:38:28 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405108, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:28.302590 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.276100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:58041 (local address 127.30.33.62:45361)
0810 23:38:28.276628 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:28.276633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:28.277992 (+ 1359us) server_negotiation.cc:407] Connection header received
0810 23:38:28.281008 (+ 3016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:28.281016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:28.281120 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:28.281274 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:28.284149 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.285088 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.288035 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.288731 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.288884 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.294508 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:28.294517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:28.294524 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:28.294582 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:28.298025 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.299406 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.299413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.299418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.299639 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.300416 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.300424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.300428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.300771 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:28.300971 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:28.301873 (+ 902us) server_negotiation.cc:299] Negotiation successful
0810 23:38:28.302161 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
I20260810 23:38:28.304005 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.276836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45361 (local address 127.30.33.1:58041)
0810 23:38:28.277781 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:28.277835 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:28.280598 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:28.281625 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:28.281639 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:28.282701 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:28.283875 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.283920 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.285317 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.285324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.287775 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.287798 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.289024 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.289030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.289262 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.290125 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:28.290161 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:28.294159 (+ 3998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:28.298265 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.298280 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.298303 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.299214 (+ 911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.299803 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.299814 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.299818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.300248 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.300933 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:28.300943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:28.301393 (+ 450us) client_negotiation.cc:769] Sending connection context
0810 23:38:28.301703 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 23:38:28.302307 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":370,"threads_started":1}
I20260810 23:38:28.329499 1852 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43427
I20260810 23:38:28.329531 2102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43427 every 8 connection(s)
I20260810 23:38:28.330960 1852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:28.332028 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 1852
I20260810 23:38:28.332247 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:28.333060 30852 external_mini_cluster.cc:1428] Setting key c9dc4bb0bc6e7a59b76a257e55bb152e
I20260810 23:38:28.333686 2105 heartbeater.cc:347] Connected to a master server at 127.30.33.62:45361
I20260810 23:38:28.333968 2105 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:28.334895 2105 heartbeater.cc:511] Master 127.30.33.62:45361 requested a full tablet report, sending...
I20260810 23:38:28.337383 1729 ts_manager.cc:194] Registered new tserver with Master: f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427)
I20260810 23:38:28.337960 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.322609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:45023 (local address 127.30.33.62:45361)
0810 23:38:28.322852 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:28.322861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:28.323521 (+ 660us) server_negotiation.cc:407] Connection header received
0810 23:38:28.323691 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:28.323695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:28.323776 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:28.323872 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:28.325260 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.326018 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.328205 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.328825 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.328957 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.330964 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:28.330970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:28.330973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:28.331009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:28.334731 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.335619 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.335626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.335629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.335816 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.336273 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.336279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.336282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.336660 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:28.336820 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:28.337330 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 23:38:28.337591 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:38:28.337989 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.322533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45361 (local address 127.30.33.1:45023)
0810 23:38:28.323323 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:28.323349 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:28.323564 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:28.323928 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:28.323933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:28.324606 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:28.325057 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.325074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.326184 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.326190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.328020 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.328040 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.329051 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.329062 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.329228 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.329876 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:28.329895 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:28.330730 (+ 835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:28.334874 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.334884 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.334889 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.335431 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.335942 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.335947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.335951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.336127 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.336799 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:28.336804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:28.337021 (+ 217us) client_negotiation.cc:769] Sending connection context
0810 23:38:28.337176 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 23:38:28.337560 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":414,"threads_started":1}
I20260810 23:38:28.339128 1729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:58041
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:28.386382 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45361
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:28.732707 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:28.733008 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:28.733150 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:28.741111 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:28.741250 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:28.741379 2115 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:38:28.741451 2115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:38:28.741561 2115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:28.752388 2115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:28.753804 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:28.755195 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:28.768775 2122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:28.768842 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:28.768750 2123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:28.769703 2115 server_base.cc:1034] running on GCE node
I20260810 23:38:28.770459 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:28.771535 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:28.772802 2115 hybrid_clock.cc:648] HybridClock initialized: now 1786405108772723 us; error 66 us; skew 500 ppm
I20260810 23:38:28.773167 2115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:28 dist-test-slave-pww0 krb5kdc[1654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405108, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:28.780961 2115 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:28.782635 2115 webserver.cc:460] Webserver started at http://127.30.33.2:37335/ using document root <none> and password file <none>
I20260810 23:38:28.783404 2115 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:28.783572 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:28.783901 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:28.787220 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "c54f44277f9d417989bec98edc9e3397"
format_stamp: "Formatted at 2026-08-10 23:38:28 on dist-test-slave-pww0"
server_key: "bef539e070ea63c5fdaddec2fa9485ea"
server_key_iv: "2d3e82172e4decf01c1fdd230f415a0b"
server_key_version: "encryptionkey@0"
I20260810 23:38:28.788128 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "c54f44277f9d417989bec98edc9e3397"
format_stamp: "Formatted at 2026-08-10 23:38:28 on dist-test-slave-pww0"
server_key: "bef539e070ea63c5fdaddec2fa9485ea"
server_key_iv: "2d3e82172e4decf01c1fdd230f415a0b"
server_key_version: "encryptionkey@0"
I20260810 23:38:28.795212 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 23:38:28.799086 2148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:28.800697 2115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:28.800865 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "c54f44277f9d417989bec98edc9e3397"
format_stamp: "Formatted at 2026-08-10 23:38:28 on dist-test-slave-pww0"
server_key: "bef539e070ea63c5fdaddec2fa9485ea"
server_key_iv: "2d3e82172e4decf01c1fdd230f415a0b"
server_key_version: "encryptionkey@0"
I20260810 23:38:28.801059 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:28.820189 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:28.824383 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:28.824653 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:28.827481 2115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:28.827633 2115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:28.827773 2115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:28.827865 2115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:38:28 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405108, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:28.852988 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.828212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:37567 (local address 127.30.33.62:45361)
0810 23:38:28.828417 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:28.828424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:28.830174 (+ 1750us) server_negotiation.cc:407] Connection header received
0810 23:38:28.832500 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:28.832506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:28.832603 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:28.832695 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:28.835110 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.836343 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.838849 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.839609 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.839761 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.845023 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:28.845032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:28.845038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:28.845088 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:28.848672 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.849846 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.849852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.849857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.850036 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.850750 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.850756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.850759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.851188 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:28.851393 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:28.852374 (+ 981us) server_negotiation.cc:299] Negotiation successful
0810 23:38:28.852627 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:38:28.854269 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.828979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45361 (local address 127.30.33.2:37567)
0810 23:38:28.829990 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:28.830032 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:28.832149 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:28.833017 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:28.833033 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:28.833915 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:28.834841 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.834879 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.836560 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.836569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.838653 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.838670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.839916 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.839924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.840153 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.841041 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:28.841079 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:28.844703 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:28.848879 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.848890 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.848914 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.849697 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.850230 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.850236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.850240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.850590 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.851349 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:28.851356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:28.851873 (+ 517us) client_negotiation.cc:769] Sending connection context
0810 23:38:28.852265 (+ 392us) client_negotiation.cc:240] Negotiation successful
0810 23:38:28.852915 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":402,"threads_started":1}
I20260810 23:38:28.877908 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.863443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:34119 (local address 127.30.33.62:45361)
0810 23:38:28.863713 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:28.863721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:28.864506 (+ 785us) server_negotiation.cc:407] Connection header received
0810 23:38:28.864692 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:28.864697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:28.864762 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:28.864847 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:28.865843 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.866613 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.868148 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.868881 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.869045 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.871057 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:28.871066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:28.871071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:28.871116 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:28.874948 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.875816 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.875822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.875825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.875985 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:28.876370 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:28.876374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:28.876376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:28.876758 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:28.876902 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:28.877338 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 23:38:28.877567 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:38:28.878026 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:28.863329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45361 (local address 127.30.33.2:34119)
0810 23:38:28.864387 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:28.864406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:28.864563 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:28.864892 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:28.864896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:28.865383 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:28.865683 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.865693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.866775 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.866779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.867963 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:28.867983 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:28.869090 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:28.869095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:28.869237 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:28.869894 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:28.869913 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:28.870843 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:28.875122 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.875128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.875133 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.875675 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.876103 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:28.876109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:28.876113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:28.876252 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:28.876879 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:28.876884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:28.877077 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 23:38:28.877194 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:38:28.877568 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":667,"threads_started":1}
I20260810 23:38:28.882172 2115 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:36771
I20260810 23:38:28.882229 2384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:36771 every 8 connection(s)
I20260810 23:38:28.883637 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:28.885701 2385 heartbeater.cc:347] Connected to a master server at 127.30.33.62:45361
I20260810 23:38:28.885924 2385 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:28.886673 2385 heartbeater.cc:511] Master 127.30.33.62:45361 requested a full tablet report, sending...
I20260810 23:38:28.888232 1726 ts_manager.cc:194] Registered new tserver with Master: c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771)
I20260810 23:38:28.889166 1726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:37567
I20260810 23:38:28.891006 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 2115
I20260810 23:38:28.891211 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:28.891829 30852 external_mini_cluster.cc:1428] Setting key 94df13ca5ac049efd787f4e8d0beafc0
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:28.940765 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45361
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:29.285880 2396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:29.286195 2396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:29.286332 2396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:29.293890 2396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:29.294037 2396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:29.294170 2396 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:38:29.294241 2396 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:38:29.294350 2396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:29.304458 2396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:29.305881 2396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:29.307283 2396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:29.320637 2411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:29.320662 2406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:29.321254 2414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:29.321470 2396 server_base.cc:1034] running on GCE node
I20260810 23:38:29.322204 2396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:29.323175 2396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:29.324402 2396 hybrid_clock.cc:648] HybridClock initialized: now 1786405109324329 us; error 58 us; skew 500 ppm
I20260810 23:38:29.324761 2396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:29 dist-test-slave-pww0 krb5kdc[1654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405109, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:29.328074 2396 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:29.329931 2396 webserver.cc:460] Webserver started at http://127.30.33.3:43623/ using document root <none> and password file <none>
I20260810 23:38:29.330940 2396 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:29.331085 2396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:29.331530 2396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:29.335985 2396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "d0f40688db8f4fb59a50f3f08710c6ef"
format_stamp: "Formatted at 2026-08-10 23:38:29 on dist-test-slave-pww0"
server_key: "2533b49aa9f199aaac3c0c0f03a2d3f2"
server_key_iv: "7dbf31a53815db582cd2c18e8bd1e4e9"
server_key_version: "encryptionkey@0"
I20260810 23:38:29.336786 2396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "d0f40688db8f4fb59a50f3f08710c6ef"
format_stamp: "Formatted at 2026-08-10 23:38:29 on dist-test-slave-pww0"
server_key: "2533b49aa9f199aaac3c0c0f03a2d3f2"
server_key_iv: "7dbf31a53815db582cd2c18e8bd1e4e9"
server_key_version: "encryptionkey@0"
I20260810 23:38:29.342504 2396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:38:29.343365 2105 heartbeater.cc:503] Master 127.30.33.62:45361 was elected leader, sending a full tablet report...
I20260810 23:38:29.346673 2427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:29.348166 2396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:29.348373 2396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "d0f40688db8f4fb59a50f3f08710c6ef"
format_stamp: "Formatted at 2026-08-10 23:38:29 on dist-test-slave-pww0"
server_key: "2533b49aa9f199aaac3c0c0f03a2d3f2"
server_key_iv: "7dbf31a53815db582cd2c18e8bd1e4e9"
server_key_version: "encryptionkey@0"
I20260810 23:38:29.348588 2396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:29.373447 2396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:29.378149 2396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:29.378463 2396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:29.381341 2396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:29.381462 2396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:29.381582 2396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:29.381661 2396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:38:29 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405109, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:29.406883 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.382035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:42849 (local address 127.30.33.62:45361)
0810 23:38:29.382619 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.382626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.383920 (+ 1294us) server_negotiation.cc:407] Connection header received
0810 23:38:29.385997 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.386005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.386108 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.386237 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:29.388677 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.389990 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.392069 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.392765 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.392916 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.398903 (+ 5987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:29.398910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:29.398913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:29.398956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:29.402349 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.403512 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.403520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.403542 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.403748 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.404362 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.404369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.404373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.404807 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:29.404984 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.406128 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.406422 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":336,"threads_started":1}
I20260810 23:38:29.408213 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.382726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45361 (local address 127.30.33.3:42849)
0810 23:38:29.383718 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.383772 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.385661 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.386597 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.386610 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.387478 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:29.388411 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.388448 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.390244 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.390258 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.391805 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.391825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.393072 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.393081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.393499 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.394448 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:29.394486 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:29.398506 (+ 4020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:29.402540 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:29.402554 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:29.402579 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:29.403321 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:29.403904 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:29.403908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:29.403911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:29.404190 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:29.404988 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:29.405002 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:29.405639 (+ 637us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.405972 (+ 333us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.406631 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":374,"threads_started":1}
I20260810 23:38:29.428501 2396 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:35915
I20260810 23:38:29.428512 2648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:35915 every 8 connection(s)
I20260810 23:38:29.430111 2396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:29.432538 2649 heartbeater.cc:347] Connected to a master server at 127.30.33.62:45361
I20260810 23:38:29.432786 2649 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:29.433638 2649 heartbeater.cc:511] Master 127.30.33.62:45361 requested a full tablet report, sending...
I20260810 23:38:29.434422 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 2396
I20260810 23:38:29.434600 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:29.435189 30852 external_mini_cluster.cc:1428] Setting key 0f199eb083dbb380861626252988f9d8
I20260810 23:38:29.435261 1726 ts_manager.cc:194] Registered new tserver with Master: d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915)
I20260810 23:38:29.436386 1726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:42849
I20260810 23:38:29.438434 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:38:29.439759 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.415160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45361 (local address 127.30.33.3:49031)
0810 23:38:29.423968 (+ 8808us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.423998 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.424162 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.424557 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.424560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.425028 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:29.425329 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.425340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.426854 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.426859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.428064 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.428085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.429080 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.429085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.429227 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.429815 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:29.429833 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:29.430805 (+ 972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:29.434834 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:29.434839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:29.434844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:29.435365 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:29.435858 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:29.435864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:29.435867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:29.436021 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:29.436832 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:29.436837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:29.437042 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.439116 (+ 2074us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.439392 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8637,"thread_start_us":8482,"threads_started":1}
I20260810 23:38:29.439874 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.415149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:49031 (local address 127.30.33.62:45361)
0810 23:38:29.415350 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.415356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.424136 (+ 8780us) server_negotiation.cc:407] Connection header received
0810 23:38:29.424280 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.424287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.424411 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.424542 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:29.425478 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.426668 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.428255 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.428897 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.429029 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.431089 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:29.431098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:29.431104 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:29.431153 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:29.434686 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.435530 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.435535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.435538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.435703 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.436135 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.436141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.436144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.436560 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:29.436718 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.439266 (+ 2548us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.439548 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 23:38:29 dist-test-slave-pww0 krb5kdc[1654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:38:29 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:29.495148 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.480905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.30.33.62:45361)
0810 23:38:29.481146 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.481154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.481381 (+ 227us) server_negotiation.cc:407] Connection header received
0810 23:38:29.481587 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.481592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.481695 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.481788 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:29.483195 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.483994 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.485275 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.485705 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.485820 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.489869 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:29.489876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:29.489880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:29.489922 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:29.492574 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.493325 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.493330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.493333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.493479 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.493838 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.493844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.493847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.494165 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:29.494401 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.494671 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.494874 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:38:29.499760 1726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:29.507398 1726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:29.542341 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.520678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.30.33.1:43427)
0810 23:38:29.534141 (+ 13463us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.534147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.534174 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:38:29.534270 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.534277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.534478 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.534679 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:29.535500 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.537021 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.539557 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.541087 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.541217 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.541439 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.541837 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.542017 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13315,"thread_start_us":13163,"threads_started":1}
I20260810 23:38:29.542445 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.520418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43427 (local address 127.0.0.1:35358)
0810 23:38:29.533170 (+ 12752us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.533242 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.533513 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.534720 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.534729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.534765 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:29.535235 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.535260 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.537147 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.537152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.539312 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.539325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.541308 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.541315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.541565 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.541621 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.541766 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.541973 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12413,"mutex_wait_us":53,"thread_start_us":12677,"threads_started":1}
I20260810 23:38:29.545128 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.523372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:35915 (local address 127.0.0.1:42788)
0810 23:38:29.536916 (+ 13544us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.536933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.537137 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.537815 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.537819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.537845 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:29.538314 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.538331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.540308 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.540313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.542348 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.542363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.544492 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.544496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.544624 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.544646 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.544718 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.544828 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13427,"mutex_wait_us":9,"thread_start_us":12304,"threads_started":1}
I20260810 23:38:29.545320 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.523693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.30.33.3:35915)
0810 23:38:29.535498 (+ 11805us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.535508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.537032 (+ 1524us) server_negotiation.cc:407] Connection header received
0810 23:38:29.537347 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.537356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.537605 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.537808 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:29.538584 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.540113 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.542561 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.543970 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.544072 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.544308 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.544801 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.544987 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11582,"thread_start_us":11382,"threads_started":1}
I20260810 23:38:29.546666 1971 tablet_service.cc:1467] Processing CreateTablet for tablet eeceb1db8c3e44d88a71b673924c6814 (DEFAULT_TABLE table=test-table [id=a26d2788620040fa8604fd00ffd6c2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:29.548102 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.520889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.30.33.2:36771)
0810 23:38:29.532126 (+ 11237us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.532139 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.536141 (+ 4002us) server_negotiation.cc:407] Connection header received
0810 23:38:29.536450 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.536458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.536740 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.536895 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:29.537678 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.539158 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.541682 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.543030 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.543169 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.543375 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.547572 (+ 4197us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.547806 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10983,"thread_start_us":11273,"threads_started":1}
I20260810 23:38:29.548130 1971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeceb1db8c3e44d88a71b673924c6814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:29.548918 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.520632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:36771 (local address 127.0.0.1:55584)
0810 23:38:29.536004 (+ 15372us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.536025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.536256 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.536980 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.536985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.537013 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:29.537505 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.537515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.539312 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.539317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.541525 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.541538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.543551 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.543558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.543715 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.543756 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.548339 (+ 4583us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.548501 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15217,"mutex_wait_us":53,"thread_start_us":16040,"threads_started":1}
I20260810 23:38:29.548767 2522 tablet_service.cc:1467] Processing CreateTablet for tablet eeceb1db8c3e44d88a71b673924c6814 (DEFAULT_TABLE table=test-table [id=a26d2788620040fa8604fd00ffd6c2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:29.550343 2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeceb1db8c3e44d88a71b673924c6814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:29.552798 2247 tablet_service.cc:1467] Processing CreateTablet for tablet eeceb1db8c3e44d88a71b673924c6814 (DEFAULT_TABLE table=test-table [id=a26d2788620040fa8604fd00ffd6c2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:29.554446 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeceb1db8c3e44d88a71b673924c6814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:29.563002 2685 tablet_bootstrap.cc:492] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Bootstrap starting.
I20260810 23:38:29.566838 2685 tablet_bootstrap.cc:654] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:29.567996 2685 log.cc:824] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:29.569626 2688 tablet_bootstrap.cc:492] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Bootstrap starting.
I20260810 23:38:29.571841 2685 tablet_bootstrap.cc:492] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: No bootstrap required, opened a new log
I20260810 23:38:29.572157 2685 ts_tablet_manager.cc:1397] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:38:29.574525 2688 tablet_bootstrap.cc:654] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:29.574569 2694 tablet_bootstrap.cc:492] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Bootstrap starting.
I20260810 23:38:29.575999 2688 log.cc:824] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:29.578262 2694 tablet_bootstrap.cc:654] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:29.579775 2688 tablet_bootstrap.cc:492] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: No bootstrap required, opened a new log
I20260810 23:38:29.579607 2694 log.cc:824] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:29.580094 2688 ts_tablet_manager.cc:1397] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 23:38:29.582425 2685 raft_consensus.cc:348] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.582918 2685 raft_consensus.cc:374] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:29.583020 2685 raft_consensus.cc:729] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Initialized, Role: FOLLOWER
I20260810 23:38:29.583751 2685 consensus_queue.cc:260] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.587405 2649 heartbeater.cc:503] Master 127.30.33.62:45361 was elected leader, sending a full tablet report...
I20260810 23:38:29.588294 2685 ts_tablet_manager.cc:1428] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260810 23:38:29.590596 2694 tablet_bootstrap.cc:492] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: No bootstrap required, opened a new log
I20260810 23:38:29.590454 2688 raft_consensus.cc:348] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.590867 2688 raft_consensus.cc:374] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:29.590937 2694 ts_tablet_manager.cc:1397] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20260810 23:38:29.590981 2688 raft_consensus.cc:729] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66fdbe8480e4284a0f6f53a20232801, State: Initialized, Role: FOLLOWER
I20260810 23:38:29.591849 2688 consensus_queue.cc:260] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.594377 2688 ts_tablet_manager.cc:1428] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260810 23:38:29.600073 2694 raft_consensus.cc:348] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.600485 2694 raft_consensus.cc:374] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:29.600587 2694 raft_consensus.cc:729] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54f44277f9d417989bec98edc9e3397, State: Initialized, Role: FOLLOWER
I20260810 23:38:29.601189 2694 consensus_queue.cc:260] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.602386 2385 heartbeater.cc:503] Master 127.30.33.62:45361 was elected leader, sending a full tablet report...
I20260810 23:38:29.603021 2694 ts_tablet_manager.cc:1428] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260810 23:38:29.636226 2386 tablet.cc:2369] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:38:29.685058 2650 tablet.cc:2369] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:38:29.834656 2106 tablet.cc:2369] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:38:29.869343 2701 raft_consensus.cc:482] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:29.869694 2701 raft_consensus.cc:504] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.871582 2701 leader_election.cc:288] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915)
I20260810 23:38:29.878795 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.872286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:49685 (local address 127.30.33.3:35915)
0810 23:38:29.872490 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.872497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.873186 (+ 689us) server_negotiation.cc:407] Connection header received
0810 23:38:29.873420 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.873425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.873484 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.873559 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:29.874301 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.875140 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.876996 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.877808 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.877915 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.878032 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.878374 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.878520 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:38:29.878859 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.872049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:43221 (local address 127.30.33.1:43427)
0810 23:38:29.872303 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.872310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.873157 (+ 847us) server_negotiation.cc:407] Connection header received
0810 23:38:29.873361 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.873365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.873428 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.873504 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:29.874137 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.874926 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.876928 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.877825 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.877958 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.878060 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.878486 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.878621 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:38:29.878964 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.871899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:35915 (local address 127.30.33.2:49685)
0810 23:38:29.873075 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.873090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.873283 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.873584 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.873589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.873612 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:29.874115 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.874130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.875294 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.875298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.876821 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.876837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.878039 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.878046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.878202 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.878235 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.878312 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.878514 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":19,"spinlock_wait_cycles":46848,"thread_start_us":599,"threads_started":1}
I20260810 23:38:29.879359 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.871889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43427 (local address 127.30.33.2:43221)
0810 23:38:29.873026 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.873047 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.873214 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.873535 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.873539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.873565 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:29.873955 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.873965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.875055 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.875061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.876756 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.876770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.878124 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.878131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.878262 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.878286 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.878434 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.878615 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":32,"thread_start_us":872,"threads_started":1}
I20260810 23:38:29.880045 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeceb1db8c3e44d88a71b673924c6814" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" is_pre_election: true
I20260810 23:38:29.880396 2022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeceb1db8c3e44d88a71b673924c6814" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801" is_pre_election: true
I20260810 23:38:29.880539 2554 raft_consensus.cc:2454] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 0.
I20260810 23:38:29.880869 2022 raft_consensus.cc:2454] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 0.
I20260810 23:38:29.881184 2153 leader_election.cc:302] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c54f44277f9d417989bec98edc9e3397, d0f40688db8f4fb59a50f3f08710c6ef; no voters:
I20260810 23:38:29.881656 2701 raft_consensus.cc:2781] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:29.881865 2701 raft_consensus.cc:482] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:29.881976 2701 raft_consensus.cc:3037] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:29.884470 2701 raft_consensus.cc:504] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.885115 2701 leader_election.cc:288] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 election: Requested vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915)
I20260810 23:38:29.885665 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeceb1db8c3e44d88a71b673924c6814" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f40688db8f4fb59a50f3f08710c6ef"
I20260810 23:38:29.885665 2022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeceb1db8c3e44d88a71b673924c6814" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801"
I20260810 23:38:29.885900 2554 raft_consensus.cc:3037] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:29.885911 2022 raft_consensus.cc:3037] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:29.889253 2022 raft_consensus.cc:2454] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 1.
I20260810 23:38:29.889261 2554 raft_consensus.cc:2454] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 1.
I20260810 23:38:29.889784 2149 leader_election.cc:302] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c54f44277f9d417989bec98edc9e3397, f66fdbe8480e4284a0f6f53a20232801; no voters:
I20260810 23:38:29.890166 2701 raft_consensus.cc:2781] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:29.890686 2701 raft_consensus.cc:686] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 1 LEADER]: Becoming Leader. State: Replica: c54f44277f9d417989bec98edc9e3397, State: Running, Role: LEADER
I20260810 23:38:29.891258 2701 consensus_queue.cc:237] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.896760 1729 catalog_manager.cc:5495] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 reported cstate change: term changed from 0 to 1, leader changed from <none> to c54f44277f9d417989bec98edc9e3397 (127.30.33.2). New cstate: current_term: 1 leader_uuid: "c54f44277f9d417989bec98edc9e3397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:38:29 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405107, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:29.964020 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.950154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45361 (local address 127.0.0.1:35508)
0810 23:38:29.950898 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:29.950914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:29.951057 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:29.951392 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:29.951397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:29.952161 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:29.952427 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.952437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.953511 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.953515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.954694 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:29.954707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.955492 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.955499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:29.955615 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.956235 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:29.956255 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:29.958578 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:29.961542 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:29.961555 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:29.961561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:29.961987 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:29.962395 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:29.962399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:29.962401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:29.962543 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:29.963059 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:29.963066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:29.963226 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 23:38:29.963385 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:38:29.963667 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":452,"threads_started":1}
I20260810 23:38:29.964058 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:29.950125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.30.33.62:45361)
0810 23:38:29.950317 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:29.950322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:29.951001 (+ 679us) server_negotiation.cc:407] Connection header received
0810 23:38:29.951169 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:29.951172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:29.951247 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:29.951323 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:29.952553 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.953389 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.954833 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:29.955298 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:29.955390 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:29.958757 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:29.958763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:29.958766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:29.958797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:29.961408 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.962129 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.962134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.962136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.962290 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:29.962651 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:29.962655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:29.962657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:29.962941 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:29.963054 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:29.963507 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 23:38:29.963795 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":56}
I20260810 23:38:29.970050 1729 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:35508:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:38:29.971217 1729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:29.978375 2247 tablet_service.cc:1467] Processing CreateTablet for tablet ae28057dce864b49a091e2582b98508c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b41f6f812ab64bfcbcebde3c7fd498d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:38:29.978211 1971 tablet_service.cc:1467] Processing CreateTablet for tablet ae28057dce864b49a091e2582b98508c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b41f6f812ab64bfcbcebde3c7fd498d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:38:29.979327 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae28057dce864b49a091e2582b98508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:29.979327 1971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae28057dce864b49a091e2582b98508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:29.984169 2522 tablet_service.cc:1467] Processing CreateTablet for tablet ae28057dce864b49a091e2582b98508c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b41f6f812ab64bfcbcebde3c7fd498d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:38:29.985140 2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae28057dce864b49a091e2582b98508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:29.986871 2694 tablet_bootstrap.cc:492] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: Bootstrap starting.
I20260810 23:38:29.987722 2688 tablet_bootstrap.cc:492] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: Bootstrap starting.
I20260810 23:38:29.988696 2694 tablet_bootstrap.cc:654] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:29.989049 2688 tablet_bootstrap.cc:654] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:29.991124 2688 tablet_bootstrap.cc:492] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: No bootstrap required, opened a new log
I20260810 23:38:29.991285 2688 ts_tablet_manager.cc:1397] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:38:29.991982 2688 raft_consensus.cc:348] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.992230 2688 raft_consensus.cc:374] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:29.992338 2688 raft_consensus.cc:729] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66fdbe8480e4284a0f6f53a20232801, State: Initialized, Role: FOLLOWER
I20260810 23:38:29.992368 2685 tablet_bootstrap.cc:492] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: Bootstrap starting.
I20260810 23:38:29.992568 2688 consensus_queue.cc:260] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.993016 2700 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:29.993414 2700 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:29.994189 2685 tablet_bootstrap.cc:654] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:29.994262 2694 tablet_bootstrap.cc:492] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: No bootstrap required, opened a new log
I20260810 23:38:29.994419 2694 ts_tablet_manager.cc:1397] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 23:38:29.995097 2694 raft_consensus.cc:348] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.995342 2694 raft_consensus.cc:374] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:29.995440 2694 raft_consensus.cc:729] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54f44277f9d417989bec98edc9e3397, State: Initialized, Role: FOLLOWER
I20260810 23:38:29.995739 2694 consensus_queue.cc:260] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.996178 2701 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:29.996466 2701 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:29.996609 2685 tablet_bootstrap.cc:492] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: No bootstrap required, opened a new log
I20260810 23:38:29.996773 2685 ts_tablet_manager.cc:1397] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:38:29.996928 2694 ts_tablet_manager.cc:1428] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:38:29.994571 2688 ts_tablet_manager.cc:1428] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:38:29.997542 2685 raft_consensus.cc:348] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.997793 2685 raft_consensus.cc:374] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:29.997898 2685 raft_consensus.cc:729] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Initialized, Role: FOLLOWER
I20260810 23:38:29.998116 2685 consensus_queue.cc:260] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:29.998713 2685 ts_tablet_manager.cc:1428] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 23:38:29.998620 2699 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:29.998950 2699 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:30.029590 2699 raft_consensus.cc:482] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:30.029816 2699 raft_consensus.cc:504] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.031502 2699 leader_election.cc:288] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771)
I20260810 23:38:30.039997 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.031891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:33521 (local address 127.30.33.2:36771)
0810 23:38:30.032104 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.032110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.032755 (+ 645us) server_negotiation.cc:407] Connection header received
0810 23:38:30.032947 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.032953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.033039 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.033146 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:30.033730 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.034893 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.037748 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.039010 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.039149 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.039274 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.039574 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.039725 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:38:30.040050 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.031801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:36771 (local address 127.30.33.3:33521)
0810 23:38:30.032617 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:30.032639 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:30.032808 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:30.033172 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:30.033178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:30.033204 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:30.033555 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.033569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.035051 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.035057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.037576 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.037595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.039185 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.039192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.039380 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.039414 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:38:30.039538 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:38:30.039653 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":158,"thread_start_us":479,"threads_started":1}
I20260810 23:38:30.040411 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.031948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43427 (local address 127.30.33.3:41251)
0810 23:38:30.032689 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:30.032708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:30.032808 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:30.033172 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:30.033178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:30.033285 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:30.033683 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.033696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.035052 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.035056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.037576 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.037595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.039495 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.039503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.039692 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.039726 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:38:30.039833 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:38:30.039969 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":145,"spinlock_wait_cycles":25984,"thread_start_us":341,"threads_started":1}
I20260810 23:38:30.040728 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.031958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:41251 (local address 127.30.33.1:43427)
0810 23:38:30.032132 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.032137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.032804 (+ 667us) server_negotiation.cc:407] Connection header received
0810 23:38:30.032947 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.032953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.033057 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.033148 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:30.033856 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.034928 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.037757 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.039010 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.039148 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.039261 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.039904 (+ 643us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.040051 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:38:30.041105 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28057dce864b49a091e2582b98508c" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54f44277f9d417989bec98edc9e3397" is_pre_election: true
I20260810 23:38:30.041347 2022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28057dce864b49a091e2582b98508c" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801" is_pre_election: true
I20260810 23:38:30.041586 2022 raft_consensus.cc:2454] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 0.
I20260810 23:38:30.041602 2287 raft_consensus.cc:2454] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 0.
I20260810 23:38:30.042071 2428 leader_election.cc:302] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0f40688db8f4fb59a50f3f08710c6ef, f66fdbe8480e4284a0f6f53a20232801; no voters:
I20260810 23:38:30.042532 2699 raft_consensus.cc:2781] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:30.042646 2699 raft_consensus.cc:482] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:30.042742 2699 raft_consensus.cc:3037] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:30.044979 2699 raft_consensus.cc:504] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.045583 2699 leader_election.cc:288] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 election: Requested vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771)
I20260810 23:38:30.046128 2022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28057dce864b49a091e2582b98508c" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801"
I20260810 23:38:30.046139 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae28057dce864b49a091e2582b98508c" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54f44277f9d417989bec98edc9e3397"
I20260810 23:38:30.046334 2287 raft_consensus.cc:3037] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:30.046360 2022 raft_consensus.cc:3037] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:30.048446 2022 raft_consensus.cc:2454] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 1.
I20260810 23:38:30.048602 2287 raft_consensus.cc:2454] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 1.
I20260810 23:38:30.048878 2428 leader_election.cc:302] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0f40688db8f4fb59a50f3f08710c6ef, f66fdbe8480e4284a0f6f53a20232801; no voters:
I20260810 23:38:30.049199 2699 raft_consensus.cc:2781] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:30.049672 2699 raft_consensus.cc:686] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [term 1 LEADER]: Becoming Leader. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Running, Role: LEADER
I20260810 23:38:30.050220 2699 consensus_queue.cc:237] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.053507 2738 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: TxnStatusTablet state changed. Reason: New leader d0f40688db8f4fb59a50f3f08710c6ef. Latest consensus state: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:30.053875 2738 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:38:30.054644 2740 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:38:30.054903 2740 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:38:30.055430 1726 catalog_manager.cc:5495] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef reported cstate change: term changed from 0 to 1, leader changed from <none> to d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3). New cstate: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } health_report { overall_health: HEALTHY } } }
I20260810 23:38:30.084038 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.077410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.30.33.3:35915)
0810 23:38:30.077620 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.077626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.077651 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:38:30.077990 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.077996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.078092 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.078216 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:30.078674 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.079572 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.081502 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.081923 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.082020 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.082714 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.083345 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.083523 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.083691 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.083777 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:38:30.084160 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.077259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:35915 (local address 127.0.0.1:42790)
0810 23:38:30.077527 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:30.077551 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:30.077730 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:30.078273 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:30.078279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:30.078292 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:38:30.078525 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.078538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.079710 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.079716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.081334 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.081355 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.082106 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.082113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.082322 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.082417 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.083510 (+ 1093us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.083546 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:38:30.083649 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:38:30.083841 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:38:30.092813 2022 raft_consensus.cc:1261] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Refusing update from remote peer d0f40688db8f4fb59a50f3f08710c6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:30.092785 2287 raft_consensus.cc:1261] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Refusing update from remote peer d0f40688db8f4fb59a50f3f08710c6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:30.093971 2738 consensus_queue.cc:1035] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:30.094447 2699 consensus_queue.cc:1035] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:30.106060 2700 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: TxnStatusTablet state changed. Reason: New leader d0f40688db8f4fb59a50f3f08710c6ef. Latest consensus state: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:30.106410 2700 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:30.108644 2701 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: TxnStatusTablet state changed. Reason: New leader d0f40688db8f4fb59a50f3f08710c6ef. Latest consensus state: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:30.108979 2701 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:30.113150 2743 mvcc.cc:205] Tried to move back new op lower bound from 7317115330928766976 to 7317115330769948672. Current Snapshot: MvccSnapshot[applied={T|T < 7317115330928766976}]
I20260810 23:38:30.113389 2738 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:30.113637 2738 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:38:30.116133 2699 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:30.118465 2699 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P d0f40688db8f4fb59a50f3f08710c6ef: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:38:30.128347 2700 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:30.128639 2700 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P f66fdbe8480e4284a0f6f53a20232801: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:30.130199 2701 tablet_replica.cc:439] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } } }
I20260810 23:38:30.130540 2701 tablet_replica.cc:442] T ae28057dce864b49a091e2582b98508c P c54f44277f9d417989bec98edc9e3397: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:38:30.137825 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.125335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.30.33.2:36771)
0810 23:38:30.125518 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.125524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.128830 (+ 3306us) server_negotiation.cc:407] Connection header received
0810 23:38:30.129444 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.129450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.129595 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.129752 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:30.133405 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.134180 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.135602 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.136020 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.136133 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.136580 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.137137 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.137282 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.137381 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.137473 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:38:30.156001 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.149128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:52011 (local address 127.30.33.3:35915)
0810 23:38:30.149311 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.149316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.149781 (+ 465us) server_negotiation.cc:407] Connection header received
0810 23:38:30.150012 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.150018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.150112 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.150202 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:30.150890 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.151768 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.153927 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.154496 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.154657 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.155293 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.155417 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.155543 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.155679 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.155762 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:38:30.156241 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.148986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:35915 (local address 127.30.33.2:52011)
0810 23:38:30.149639 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:30.149685 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:30.149865 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:30.150241 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:30.150255 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:30.150270 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:38:30.150658 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.150672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.151934 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.151941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.153694 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.153714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.154733 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.154748 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.154942 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.155035 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.155547 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.155560 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:38:30.155651 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:38:30.155828 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":370,"threads_started":1}
I20260810 23:38:30.167078 2022 raft_consensus.cc:1261] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Refusing update from remote peer c54f44277f9d417989bec98edc9e3397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:30.167444 2554 raft_consensus.cc:1261] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Refusing update from remote peer c54f44277f9d417989bec98edc9e3397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:30.167752 2708 consensus_queue.cc:1035] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:30.168257 2701 consensus_queue.cc:1035] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:30.219215 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.212497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:45803 (local address 127.30.33.2:36771)
0810 23:38:30.212692 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.212696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.213405 (+ 709us) server_negotiation.cc:407] Connection header received
0810 23:38:30.213620 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.213626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.213709 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.213796 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:30.214366 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.215099 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.216934 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.217624 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.217795 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.218560 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.218681 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.218762 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.218925 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.218998 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:38:30.219555 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.212440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:36771 (local address 127.30.33.3:45803)
0810 23:38:30.213289 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:30.213315 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:30.213465 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:30.213832 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:30.213837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:30.213849 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:38:30.214201 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.214212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.215246 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.215252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.216723 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.216738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.217968 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.217976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.218196 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.218293 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.218800 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.218812 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 23:38:30.218910 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:38:30.219080 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":448,"threads_started":1}
I20260810 23:38:30.263222 1726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:30.263603 1726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:30.268205 1726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Sending DeleteTablet for 3 replicas of tablet eeceb1db8c3e44d88a71b673924c6814
I20260810 23:38:30.269650 1971 tablet_service.cc:1514] Processing DeleteTablet for tablet eeceb1db8c3e44d88a71b673924c6814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:30 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:35358
I20260810 23:38:30.269996 2522 tablet_service.cc:1514] Processing DeleteTablet for tablet eeceb1db8c3e44d88a71b673924c6814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:30 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:42788
I20260810 23:38:30.271184 2247 tablet_service.cc:1514] Processing DeleteTablet for tablet eeceb1db8c3e44d88a71b673924c6814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:30 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:55584
I20260810 23:38:30.276129 2788 tablet_replica.cc:330] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: stopping tablet replica
I20260810 23:38:30.276382 1726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:38:30.276677 2788 raft_consensus.cc:2229] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:30.277402 2788 raft_consensus.cc:2258] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:38:30.277493 1726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:30.285465 2785 tablet_replica.cc:330] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: stopping tablet replica
I20260810 23:38:30.286845 2520 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 (DEFAULT_TABLE table=test-table [id=583fa988806e4b5bb9d92da21a0b33a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:30.287302 2247 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 (DEFAULT_TABLE table=test-table [id=583fa988806e4b5bb9d92da21a0b33a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:30.287801 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:30.287848 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:30.290131 2788 ts_tablet_manager.cc:1905] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:30.291392 2787 tablet_replica.cc:330] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: stopping tablet replica
I20260810 23:38:30.292028 2787 raft_consensus.cc:2229] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:30.292537 2787 raft_consensus.cc:2258] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:30.295522 2694 tablet_bootstrap.cc:492] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: Bootstrap starting.
I20260810 23:38:30.295923 2685 tablet_bootstrap.cc:492] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: Bootstrap starting.
I20260810 23:38:30.297482 2685 tablet_bootstrap.cc:654] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:30.297494 2694 tablet_bootstrap.cc:654] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:30.298156 2788 ts_tablet_manager.cc:1918] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:30.298346 2788 log.cc:1196] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/eeceb1db8c3e44d88a71b673924c6814
I20260810 23:38:30.298918 2788 ts_tablet_manager.cc:1939] T eeceb1db8c3e44d88a71b673924c6814 P c54f44277f9d417989bec98edc9e3397: Deleting consensus metadata
I20260810 23:38:30.286952 1970 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 (DEFAULT_TABLE table=test-table [id=583fa988806e4b5bb9d92da21a0b33a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:30.304090 2785 raft_consensus.cc:2229] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:30.304101 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:30.304677 2785 raft_consensus.cc:2258] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:30.304777 2787 ts_tablet_manager.cc:1905] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:30.307313 2785 ts_tablet_manager.cc:1905] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:30.311820 2688 tablet_bootstrap.cc:492] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: Bootstrap starting.
I20260810 23:38:30.313783 2688 tablet_bootstrap.cc:654] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:30.318449 2688 tablet_bootstrap.cc:492] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: No bootstrap required, opened a new log
I20260810 23:38:30.318655 2688 ts_tablet_manager.cc:1397] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260810 23:38:30.318836 1705 catalog_manager.cc:4841] TS c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771): tablet eeceb1db8c3e44d88a71b673924c6814 (table test-table [id=a26d2788620040fa8604fd00ffd6c2ac]) successfully deleted
I20260810 23:38:30.319440 2688 raft_consensus.cc:348] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.319875 2688 raft_consensus.cc:374] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:30.320012 2688 raft_consensus.cc:729] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66fdbe8480e4284a0f6f53a20232801, State: Initialized, Role: FOLLOWER
I20260810 23:38:30.320376 2694 tablet_bootstrap.cc:492] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: No bootstrap required, opened a new log
I20260810 23:38:30.320564 2694 ts_tablet_manager.cc:1397] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20260810 23:38:30.320369 2688 consensus_queue.cc:260] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.321425 2694 raft_consensus.cc:348] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.321689 2694 raft_consensus.cc:374] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:30.321779 2694 raft_consensus.cc:729] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54f44277f9d417989bec98edc9e3397, State: Initialized, Role: FOLLOWER
I20260810 23:38:30.322057 2694 consensus_queue.cc:260] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.323791 2694 ts_tablet_manager.cc:1428] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 23:38:30.327528 2785 ts_tablet_manager.cc:1918] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:30.327785 2785 log.cc:1196] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/eeceb1db8c3e44d88a71b673924c6814
I20260810 23:38:30.328308 2688 ts_tablet_manager.cc:1428] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20260810 23:38:30.328402 2785 ts_tablet_manager.cc:1939] T eeceb1db8c3e44d88a71b673924c6814 P f66fdbe8480e4284a0f6f53a20232801: Deleting consensus metadata
I20260810 23:38:30.329701 2787 ts_tablet_manager.cc:1918] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:30.329963 2787 log.cc:1196] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/eeceb1db8c3e44d88a71b673924c6814
I20260810 23:38:30.330655 2787 ts_tablet_manager.cc:1939] T eeceb1db8c3e44d88a71b673924c6814 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting consensus metadata
I20260810 23:38:30.332141 2685 tablet_bootstrap.cc:492] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: No bootstrap required, opened a new log
I20260810 23:38:30.332373 2685 ts_tablet_manager.cc:1397] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.005s
I20260810 23:38:30.332965 1700 catalog_manager.cc:4841] TS f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427): tablet eeceb1db8c3e44d88a71b673924c6814 (table test-table [id=a26d2788620040fa8604fd00ffd6c2ac]) successfully deleted
I20260810 23:38:30.333271 2685 raft_consensus.cc:348] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.333624 2685 raft_consensus.cc:374] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:30.333766 2685 raft_consensus.cc:729] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Initialized, Role: FOLLOWER
I20260810 23:38:30.334127 2685 consensus_queue.cc:260] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.335660 1701 catalog_manager.cc:4841] TS d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915): tablet eeceb1db8c3e44d88a71b673924c6814 (table test-table [id=a26d2788620040fa8604fd00ffd6c2ac]) successfully deleted
I20260810 23:38:30.336862 2685 ts_tablet_manager.cc:1428] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:38:30.587158 2699 raft_consensus.cc:482] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:30.587385 2699 raft_consensus.cc:504] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.588176 2699 leader_election.cc:288] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771)
I20260810 23:38:30.588797 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cc2ab777544a3b1c8368dbd4b1a06" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54f44277f9d417989bec98edc9e3397" is_pre_election: true
I20260810 23:38:30.588843 2022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cc2ab777544a3b1c8368dbd4b1a06" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801" is_pre_election: true
I20260810 23:38:30.589053 2287 raft_consensus.cc:2454] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 0.
I20260810 23:38:30.589134 2022 raft_consensus.cc:2454] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 0.
I20260810 23:38:30.589563 2434 leader_election.cc:302] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c54f44277f9d417989bec98edc9e3397, d0f40688db8f4fb59a50f3f08710c6ef; no voters:
I20260810 23:38:30.589834 2699 raft_consensus.cc:2781] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:30.589953 2699 raft_consensus.cc:482] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:30.590039 2699 raft_consensus.cc:3037] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:30.592123 2699 raft_consensus.cc:504] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.592679 2699 leader_election.cc:288] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 election: Requested vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771)
I20260810 23:38:30.593276 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cc2ab777544a3b1c8368dbd4b1a06" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54f44277f9d417989bec98edc9e3397"
I20260810 23:38:30.593282 2022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5cc2ab777544a3b1c8368dbd4b1a06" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801"
I20260810 23:38:30.593494 2287 raft_consensus.cc:3037] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:30.593502 2022 raft_consensus.cc:3037] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:30.595988 2287 raft_consensus.cc:2454] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 1.
I20260810 23:38:30.596536 2434 leader_election.cc:302] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c54f44277f9d417989bec98edc9e3397, d0f40688db8f4fb59a50f3f08710c6ef; no voters:
I20260810 23:38:30.596719 2022 raft_consensus.cc:2454] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 1.
I20260810 23:38:30.596827 2699 raft_consensus.cc:2781] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:30.596997 2699 raft_consensus.cc:686] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 LEADER]: Becoming Leader. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Running, Role: LEADER
I20260810 23:38:30.597328 2699 consensus_queue.cc:237] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:30.600449 1726 catalog_manager.cc:5495] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef reported cstate change: term changed from 0 to 1, leader changed from <none> to d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3). New cstate: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } health_report { overall_health: HEALTHY } } }
I20260810 23:38:30.692534 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.687194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.30.33.3:35915)
0810 23:38:30.687900 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.687906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.687923 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:38:30.687990 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.687995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.688063 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.688149 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:30.688577 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.689293 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.690660 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.691080 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.691198 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.691655 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.691771 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.691850 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.692006 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.692093 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":469,"threads_started":1}
I20260810 23:38:30.706480 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.701760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:35915 (local address 127.30.33.3:52773)
0810 23:38:30.701958 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:30.701980 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:30.702097 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:30.702365 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:30.702369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:30.702379 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:38:30.702577 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.702586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.703640 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.703646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.704717 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:30.704732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.705367 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.705373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:30.705562 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.705612 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.705957 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.705968 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:38:30.706043 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:38:30.706179 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":20}
I20260810 23:38:30.706477 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:30.701883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:52773 (local address 127.30.33.3:35915)
0810 23:38:30.702012 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:30.702018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:30.702036 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:38:30.702210 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:30.702214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:30.702273 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:30.702344 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:30.702694 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.703497 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.704850 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:30.705231 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:30.705339 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:30.705764 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:30.705870 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:30.705944 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:30.706066 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 23:38:30.706140 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":36}
I20260810 23:38:30.714442 2022 raft_consensus.cc:1261] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Refusing update from remote peer d0f40688db8f4fb59a50f3f08710c6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:30.714476 2287 raft_consensus.cc:1261] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Refusing update from remote peer d0f40688db8f4fb59a50f3f08710c6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:30.714984 2738 consensus_queue.cc:1035] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:30.715276 2699 consensus_queue.cc:1035] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:38:30.736146 2743 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:38:32.327541 1726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:32.327817 1726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:32.332732 1726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Sending DeleteTablet for 3 replicas of tablet 5c5cc2ab777544a3b1c8368dbd4b1a06
I20260810 23:38:32.333714 1970 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:32 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:35358
I20260810 23:38:32.333814 2247 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:32 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:55584
I20260810 23:38:32.334534 2865 tablet_replica.cc:330] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: stopping tablet replica
I20260810 23:38:32.334501 2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:32 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:42788
I20260810 23:38:32.334784 2865 raft_consensus.cc:2229] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:32.335070 2865 raft_consensus.cc:2258] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:32.336462 2865 ts_tablet_manager.cc:1905] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:32.343521 2865 ts_tablet_manager.cc:1918] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:38:32.343717 2865 log.cc:1196] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/5c5cc2ab777544a3b1c8368dbd4b1a06
I20260810 23:38:32.344265 2865 ts_tablet_manager.cc:1939] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P f66fdbe8480e4284a0f6f53a20232801: Deleting consensus metadata
I20260810 23:38:32.345695 1700 catalog_manager.cc:4841] TS f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427): tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 (table test-table [id=583fa988806e4b5bb9d92da21a0b33a4]) successfully deleted
I20260810 23:38:32.345852 2868 tablet_replica.cc:330] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: stopping tablet replica
I20260810 23:38:32.346055 2868 raft_consensus.cc:2229] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:32.346347 2868 raft_consensus.cc:2258] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:32.347323 2870 tablet_replica.cc:330] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: stopping tablet replica
I20260810 23:38:32.347318 2868 ts_tablet_manager.cc:1905] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:32.348030 2870 raft_consensus.cc:2229] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:32.348664 2870 raft_consensus.cc:2258] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:32.349812 2870 ts_tablet_manager.cc:1905] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:32.352975 2868 ts_tablet_manager.cc:1918] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:38:32.353196 2868 log.cc:1196] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/5c5cc2ab777544a3b1c8368dbd4b1a06
I20260810 23:38:32.353613 2868 ts_tablet_manager.cc:1939] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P c54f44277f9d417989bec98edc9e3397: Deleting consensus metadata
I20260810 23:38:32.354688 1705 catalog_manager.cc:4841] TS c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771): tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 (table test-table [id=583fa988806e4b5bb9d92da21a0b33a4]) successfully deleted
I20260810 23:38:32.356356 2870 ts_tablet_manager.cc:1918] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:38:32.356557 2870 log.cc:1196] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/5c5cc2ab777544a3b1c8368dbd4b1a06
I20260810 23:38:32.357012 2870 ts_tablet_manager.cc:1939] T 5c5cc2ab777544a3b1c8368dbd4b1a06 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting consensus metadata
I20260810 23:38:32.358130 1701 catalog_manager.cc:4841] TS d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915): tablet 5c5cc2ab777544a3b1c8368dbd4b1a06 (table test-table [id=583fa988806e4b5bb9d92da21a0b33a4]) successfully deleted
Aug 10 23:38:32 dist-test-slave-pww0 krb5kdc[1654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405112, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:38:32 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405112, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:32.400480 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:32.385571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.30.33.62:45361)
0810 23:38:32.386115 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:32.386119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:32.386143 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:38:32.386373 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:32.386378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:32.386448 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:32.386570 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:32.387992 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:32.388826 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:32.390090 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:32.390536 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:32.390635 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:32.394838 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:32.394845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:32.394849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:32.394899 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:32.397747 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:32.398618 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:32.398624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:32.398626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:32.398796 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:32.399185 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:32.399189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:32.399191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:32.399568 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:32.399775 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:32.400004 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 23:38:32.400189 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":255,"threads_started":1}
I20260810 23:38:32.403961 1726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:32.404677 1726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:32.411348 1970 tablet_service.cc:1467] Processing CreateTablet for tablet 1beb2a1267b047a1b3dc88883908f5b5 (DEFAULT_TABLE table=test-table [id=492aebd99a694f64a30d46d05dfc132c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:32.411922 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1beb2a1267b047a1b3dc88883908f5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:32.411816 2520 tablet_service.cc:1467] Processing CreateTablet for tablet 1beb2a1267b047a1b3dc88883908f5b5 (DEFAULT_TABLE table=test-table [id=492aebd99a694f64a30d46d05dfc132c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:32.411952 2247 tablet_service.cc:1467] Processing CreateTablet for tablet 1beb2a1267b047a1b3dc88883908f5b5 (DEFAULT_TABLE table=test-table [id=492aebd99a694f64a30d46d05dfc132c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:32.412269 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1beb2a1267b047a1b3dc88883908f5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:32.412341 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1beb2a1267b047a1b3dc88883908f5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:32.417559 2887 tablet_bootstrap.cc:492] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: Bootstrap starting.
I20260810 23:38:32.419404 2887 tablet_bootstrap.cc:654] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:32.420918 2891 tablet_bootstrap.cc:492] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: Bootstrap starting.
I20260810 23:38:32.421581 2887 tablet_bootstrap.cc:492] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: No bootstrap required, opened a new log
I20260810 23:38:32.421696 2889 tablet_bootstrap.cc:492] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: Bootstrap starting.
I20260810 23:38:32.421738 2887 ts_tablet_manager.cc:1397] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:38:32.422607 2887 raft_consensus.cc:348] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.422889 2887 raft_consensus.cc:374] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:32.422910 2891 tablet_bootstrap.cc:654] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:32.423003 2887 raft_consensus.cc:729] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54f44277f9d417989bec98edc9e3397, State: Initialized, Role: FOLLOWER
I20260810 23:38:32.423270 2887 consensus_queue.cc:260] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.423425 2889 tablet_bootstrap.cc:654] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:32.425733 2889 tablet_bootstrap.cc:492] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: No bootstrap required, opened a new log
I20260810 23:38:32.425957 2889 ts_tablet_manager.cc:1397] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260810 23:38:32.426856 2889 raft_consensus.cc:348] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.427196 2889 raft_consensus.cc:374] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:32.427331 2889 raft_consensus.cc:729] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66fdbe8480e4284a0f6f53a20232801, State: Initialized, Role: FOLLOWER
I20260810 23:38:32.427695 2889 consensus_queue.cc:260] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.437738 2889 ts_tablet_manager.cc:1428] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20260810 23:38:32.438632 2887 ts_tablet_manager.cc:1428] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260810 23:38:32.438899 2891 tablet_bootstrap.cc:492] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: No bootstrap required, opened a new log
I20260810 23:38:32.439090 2891 ts_tablet_manager.cc:1397] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.009s
I20260810 23:38:32.440189 2891 raft_consensus.cc:348] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.440608 2891 raft_consensus.cc:374] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:32.440794 2891 raft_consensus.cc:729] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Initialized, Role: FOLLOWER
I20260810 23:38:32.441147 2891 consensus_queue.cc:260] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.443228 2891 ts_tablet_manager.cc:1428] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:38:32.465685 2893 raft_consensus.cc:482] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:32.465953 2893 raft_consensus.cc:504] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.466780 2893 leader_election.cc:288] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915)
I20260810 23:38:32.467494 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb2a1267b047a1b3dc88883908f5b5" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801" is_pre_election: true
I20260810 23:38:32.467494 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb2a1267b047a1b3dc88883908f5b5" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" is_pre_election: true
I20260810 23:38:32.467769 2014 raft_consensus.cc:2454] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 0.
I20260810 23:38:32.467792 2554 raft_consensus.cc:2454] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 0.
I20260810 23:38:32.468302 2153 leader_election.cc:302] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c54f44277f9d417989bec98edc9e3397, d0f40688db8f4fb59a50f3f08710c6ef, f66fdbe8480e4284a0f6f53a20232801; no voters:
I20260810 23:38:32.468611 2893 raft_consensus.cc:2781] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:32.468771 2893 raft_consensus.cc:482] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:32.468863 2893 raft_consensus.cc:3037] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:32.471285 2893 raft_consensus.cc:504] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.471932 2893 leader_election.cc:288] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 election: Requested vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915)
I20260810 23:38:32.472589 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb2a1267b047a1b3dc88883908f5b5" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801"
I20260810 23:38:32.472589 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1beb2a1267b047a1b3dc88883908f5b5" candidate_uuid: "c54f44277f9d417989bec98edc9e3397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f40688db8f4fb59a50f3f08710c6ef"
I20260810 23:38:32.472832 2014 raft_consensus.cc:3037] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:32.472832 2554 raft_consensus.cc:3037] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:32.475944 2014 raft_consensus.cc:2454] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 1.
I20260810 23:38:32.475944 2554 raft_consensus.cc:2454] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c54f44277f9d417989bec98edc9e3397 in term 1.
I20260810 23:38:32.476540 2149 leader_election.cc:302] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c54f44277f9d417989bec98edc9e3397, d0f40688db8f4fb59a50f3f08710c6ef, f66fdbe8480e4284a0f6f53a20232801; no voters:
I20260810 23:38:32.476897 2893 raft_consensus.cc:2781] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:32.477352 2893 raft_consensus.cc:686] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 1 LEADER]: Becoming Leader. State: Replica: c54f44277f9d417989bec98edc9e3397, State: Running, Role: LEADER
I20260810 23:38:32.477670 2893 consensus_queue.cc:237] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.480582 1729 catalog_manager.cc:5495] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 reported cstate change: term changed from 0 to 1, leader changed from <none> to c54f44277f9d417989bec98edc9e3397 (127.30.33.2). New cstate: current_term: 1 leader_uuid: "c54f44277f9d417989bec98edc9e3397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } health_report { overall_health: UNKNOWN } } }
I20260810 23:38:32.514029 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:32.508808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.30.33.2:36771)
0810 23:38:32.509437 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:32.509441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:32.509460 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:38:32.509517 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:32.509520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:32.509582 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:32.509670 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:32.510119 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:32.510915 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:32.512273 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:32.512704 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:32.512820 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:32.513255 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:32.513382 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:32.513460 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:32.513621 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 23:38:32.513702 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":347,"threads_started":1}
I20260810 23:38:32.524088 2014 raft_consensus.cc:1261] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Refusing update from remote peer c54f44277f9d417989bec98edc9e3397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:32.524127 2554 raft_consensus.cc:1261] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Refusing update from remote peer c54f44277f9d417989bec98edc9e3397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:32.524667 2902 consensus_queue.cc:1035] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:32.524972 2893 consensus_queue.cc:1035] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:32.534070 2839 mvcc.cc:205] Tried to move back new op lower bound from 7317115340893798400 to 7317115340711178240. Current Snapshot: MvccSnapshot[applied={T|T < 7317115340893798400}]
I20260810 23:38:32.570936 1726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:32.571208 1726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:32.574373 1726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Sending DeleteTablet for 3 replicas of tablet 1beb2a1267b047a1b3dc88883908f5b5
I20260810 23:38:32.575344 1970 tablet_service.cc:1514] Processing DeleteTablet for tablet 1beb2a1267b047a1b3dc88883908f5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:32 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:35358
I20260810 23:38:32.575364 2247 tablet_service.cc:1514] Processing DeleteTablet for tablet 1beb2a1267b047a1b3dc88883908f5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:32 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:55584
I20260810 23:38:32.575443 2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 1beb2a1267b047a1b3dc88883908f5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:32 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:42788
I20260810 23:38:32.575693 2865 tablet_replica.cc:330] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: stopping tablet replica
I20260810 23:38:32.575778 2870 tablet_replica.cc:330] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: stopping tablet replica
I20260810 23:38:32.575809 2868 tablet_replica.cc:330] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: stopping tablet replica
I20260810 23:38:32.575901 2865 raft_consensus.cc:2229] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:32.575965 2870 raft_consensus.cc:2229] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:32.576020 2868 raft_consensus.cc:2229] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:32.576105 2865 raft_consensus.cc:2258] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:32.576169 2870 raft_consensus.cc:2258] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:32.576392 2868 raft_consensus.cc:2258] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:32.576644 1726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:32.577317 1726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:32.577448 2870 ts_tablet_manager.cc:1905] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:32.577497 2865 ts_tablet_manager.cc:1905] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:32.579224 2868 ts_tablet_manager.cc:1905] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:32.584178 2865 ts_tablet_manager.cc:1918] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:32.584460 2865 log.cc:1196] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/1beb2a1267b047a1b3dc88883908f5b5
I20260810 23:38:32.584769 2870 ts_tablet_manager.cc:1918] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:32.584996 2870 log.cc:1196] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/1beb2a1267b047a1b3dc88883908f5b5
I20260810 23:38:32.585032 2865 ts_tablet_manager.cc:1939] T 1beb2a1267b047a1b3dc88883908f5b5 P f66fdbe8480e4284a0f6f53a20232801: Deleting consensus metadata
I20260810 23:38:32.585536 2870 ts_tablet_manager.cc:1939] T 1beb2a1267b047a1b3dc88883908f5b5 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting consensus metadata
I20260810 23:38:32.586215 1700 catalog_manager.cc:4841] TS f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427): tablet 1beb2a1267b047a1b3dc88883908f5b5 (table test-table [id=492aebd99a694f64a30d46d05dfc132c]) successfully deleted
I20260810 23:38:32.587046 1701 catalog_manager.cc:4841] TS d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915): tablet 1beb2a1267b047a1b3dc88883908f5b5 (table test-table [id=492aebd99a694f64a30d46d05dfc132c]) successfully deleted
I20260810 23:38:32.587685 2868 ts_tablet_manager.cc:1918] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:38:32.587836 1970 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca45c8b3877492485a4f0a0f714c400 (DEFAULT_TABLE table=test-table [id=2717a45c84534ef6b7a418b13237980a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:32.588004 2868 log.cc:1196] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/1beb2a1267b047a1b3dc88883908f5b5
I20260810 23:38:32.588282 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca45c8b3877492485a4f0a0f714c400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:32.588217 2520 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca45c8b3877492485a4f0a0f714c400 (DEFAULT_TABLE table=test-table [id=2717a45c84534ef6b7a418b13237980a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:32.588512 2868 ts_tablet_manager.cc:1939] T 1beb2a1267b047a1b3dc88883908f5b5 P c54f44277f9d417989bec98edc9e3397: Deleting consensus metadata
I20260810 23:38:32.588485 2247 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca45c8b3877492485a4f0a0f714c400 (DEFAULT_TABLE table=test-table [id=2717a45c84534ef6b7a418b13237980a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:32.588626 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca45c8b3877492485a4f0a0f714c400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:32.588913 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca45c8b3877492485a4f0a0f714c400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:32.590238 1705 catalog_manager.cc:4841] TS c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771): tablet 1beb2a1267b047a1b3dc88883908f5b5 (table test-table [id=492aebd99a694f64a30d46d05dfc132c]) successfully deleted
I20260810 23:38:32.593823 2889 tablet_bootstrap.cc:492] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: Bootstrap starting.
I20260810 23:38:32.595124 2887 tablet_bootstrap.cc:492] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: Bootstrap starting.
I20260810 23:38:32.595407 2891 tablet_bootstrap.cc:492] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: Bootstrap starting.
I20260810 23:38:32.595582 2889 tablet_bootstrap.cc:654] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:32.596877 2887 tablet_bootstrap.cc:654] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:32.596982 2891 tablet_bootstrap.cc:654] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:32.598188 2889 tablet_bootstrap.cc:492] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: No bootstrap required, opened a new log
I20260810 23:38:32.598361 2889 ts_tablet_manager.cc:1397] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260810 23:38:32.599028 2889 raft_consensus.cc:348] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.599244 2889 raft_consensus.cc:374] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:32.599325 2889 raft_consensus.cc:729] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66fdbe8480e4284a0f6f53a20232801, State: Initialized, Role: FOLLOWER
I20260810 23:38:32.599620 2889 consensus_queue.cc:260] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.600270 2889 ts_tablet_manager.cc:1428] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:38:32.608964 2887 tablet_bootstrap.cc:492] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: No bootstrap required, opened a new log
I20260810 23:38:32.609076 2891 tablet_bootstrap.cc:492] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: No bootstrap required, opened a new log
I20260810 23:38:32.609319 2891 ts_tablet_manager.cc:1397] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20260810 23:38:32.609638 2887 ts_tablet_manager.cc:1397] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260810 23:38:32.610139 2891 raft_consensus.cc:348] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.610424 2891 raft_consensus.cc:374] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:32.610515 2891 raft_consensus.cc:729] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Initialized, Role: FOLLOWER
I20260810 23:38:32.610512 2887 raft_consensus.cc:348] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.610769 2887 raft_consensus.cc:374] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:32.610857 2887 raft_consensus.cc:729] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54f44277f9d417989bec98edc9e3397, State: Initialized, Role: FOLLOWER
I20260810 23:38:32.610796 2891 consensus_queue.cc:260] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.611107 2887 consensus_queue.cc:260] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.611388 2891 ts_tablet_manager.cc:1428] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 23:38:32.611726 2887 ts_tablet_manager.cc:1428] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 23:38:32.784808 2843 raft_consensus.cc:482] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:32.785017 2843 raft_consensus.cc:504] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.785666 2843 leader_election.cc:288] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771)
I20260810 23:38:32.786221 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca45c8b3877492485a4f0a0f714c400" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801" is_pre_election: true
I20260810 23:38:32.786312 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca45c8b3877492485a4f0a0f714c400" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54f44277f9d417989bec98edc9e3397" is_pre_election: true
I20260810 23:38:32.786469 2014 raft_consensus.cc:2454] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 0.
I20260810 23:38:32.786574 2287 raft_consensus.cc:2454] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 0.
I20260810 23:38:32.786906 2428 leader_election.cc:302] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0f40688db8f4fb59a50f3f08710c6ef, f66fdbe8480e4284a0f6f53a20232801; no voters:
I20260810 23:38:32.787214 2843 raft_consensus.cc:2781] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:32.787343 2843 raft_consensus.cc:482] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:32.787421 2843 raft_consensus.cc:3037] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:32.789916 2843 raft_consensus.cc:504] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.790498 2843 leader_election.cc:288] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 election: Requested vote from peers f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427), c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771)
I20260810 23:38:32.791049 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca45c8b3877492485a4f0a0f714c400" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66fdbe8480e4284a0f6f53a20232801"
I20260810 23:38:32.791100 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca45c8b3877492485a4f0a0f714c400" candidate_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54f44277f9d417989bec98edc9e3397"
I20260810 23:38:32.791312 2287 raft_consensus.cc:3037] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:32.791308 2014 raft_consensus.cc:3037] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:32.794390 2014 raft_consensus.cc:2454] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 1.
I20260810 23:38:32.794390 2287 raft_consensus.cc:2454] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f40688db8f4fb59a50f3f08710c6ef in term 1.
I20260810 23:38:32.794934 2434 leader_election.cc:302] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c54f44277f9d417989bec98edc9e3397, d0f40688db8f4fb59a50f3f08710c6ef; no voters:
I20260810 23:38:32.795363 2843 raft_consensus.cc:2781] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:32.795549 2843 raft_consensus.cc:686] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 LEADER]: Becoming Leader. State: Replica: d0f40688db8f4fb59a50f3f08710c6ef, State: Running, Role: LEADER
I20260810 23:38:32.795828 2843 consensus_queue.cc:237] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } }
I20260810 23:38:32.798712 1726 catalog_manager.cc:5495] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef reported cstate change: term changed from 0 to 1, leader changed from <none> to d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3). New cstate: current_term: 1 leader_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f40688db8f4fb59a50f3f08710c6ef" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35915 } health_report { overall_health: HEALTHY } } }
I20260810 23:38:32.848977 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:32.843813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.30.33.3:35915)
0810 23:38:32.844411 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:32.844415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:32.844434 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:38:32.844487 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:32.844490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:32.844552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:32.844646 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:32.845082 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:32.845862 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:32.847184 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:32.847634 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:32.847750 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:32.848213 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:32.848335 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:32.848414 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:32.848559 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 23:38:32.848646 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":342,"threads_started":1}
I20260810 23:38:32.858054 2287 raft_consensus.cc:1261] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Refusing update from remote peer d0f40688db8f4fb59a50f3f08710c6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:32.858107 2014 raft_consensus.cc:1261] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Refusing update from remote peer d0f40688db8f4fb59a50f3f08710c6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:32.858672 2858 consensus_queue.cc:1035] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54f44277f9d417989bec98edc9e3397" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:32.858952 2843 consensus_queue.cc:1035] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66fdbe8480e4284a0f6f53a20232801" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:38:32.879684 2821 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:38:34.467758 1729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:34.468003 1729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:34.472784 1729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55f9bd8eb6024fe7adec5410e6e8a1f7: Sending DeleteTablet for 3 replicas of tablet 4ca45c8b3877492485a4f0a0f714c400
I20260810 23:38:34.473832 2247 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ca45c8b3877492485a4f0a0f714c400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:34 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:55584
I20260810 23:38:34.473814 1970 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ca45c8b3877492485a4f0a0f714c400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:34 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:35358
I20260810 23:38:34.473866 2522 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ca45c8b3877492485a4f0a0f714c400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:34 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:42788
I20260810 23:38:34.475602 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1852
I20260810 23:38:34.476497 2960 tablet_replica.cc:330] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: stopping tablet replica
I20260810 23:38:34.476830 2960 raft_consensus.cc:2229] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:34.477237 2960 raft_consensus.cc:2258] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:34.478534 2960 ts_tablet_manager.cc:1905] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:34.486668 2961 tablet_replica.cc:330] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: stopping tablet replica
I20260810 23:38:34.487051 2957 tablet_replica.cc:330] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: stopping tablet replica
I20260810 23:38:34.487047 2961 raft_consensus.cc:2229] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:34.487429 2957 raft_consensus.cc:2229] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:34.487644 2961 raft_consensus.cc:2258] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:34.487916 2957 raft_consensus.cc:2258] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:34.488850 2961 ts_tablet_manager.cc:1905] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:34.489255 2957 ts_tablet_manager.cc:1905] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:34.497087 2961 ts_tablet_manager.cc:1918] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:38:34.497313 2957 ts_tablet_manager.cc:1918] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:38:34.497515 2957 log.cc:1196] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/4ca45c8b3877492485a4f0a0f714c400
I20260810 23:38:34.497526 2960 ts_tablet_manager.cc:1918] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:38:34.497828 2960 log.cc:1196] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/4ca45c8b3877492485a4f0a0f714c400
I20260810 23:38:34.498063 2957 ts_tablet_manager.cc:1939] T 4ca45c8b3877492485a4f0a0f714c400 P f66fdbe8480e4284a0f6f53a20232801: Deleting consensus metadata
I20260810 23:38:34.498502 2960 ts_tablet_manager.cc:1939] T 4ca45c8b3877492485a4f0a0f714c400 P c54f44277f9d417989bec98edc9e3397: Deleting consensus metadata
I20260810 23:38:34.499667 1700 catalog_manager.cc:4841] TS f66fdbe8480e4284a0f6f53a20232801 (127.30.33.1:43427): tablet 4ca45c8b3877492485a4f0a0f714c400 (table test-table [id=2717a45c84534ef6b7a418b13237980a]) successfully deleted
I20260810 23:38:34.499943 1705 catalog_manager.cc:4841] TS c54f44277f9d417989bec98edc9e3397 (127.30.33.2:36771): tablet 4ca45c8b3877492485a4f0a0f714c400 (table test-table [id=2717a45c84534ef6b7a418b13237980a]) successfully deleted
I20260810 23:38:34.500347 2961 log.cc:1196] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/4ca45c8b3877492485a4f0a0f714c400
I20260810 23:38:34.500756 2961 ts_tablet_manager.cc:1939] T 4ca45c8b3877492485a4f0a0f714c400 P d0f40688db8f4fb59a50f3f08710c6ef: Deleting consensus metadata
I20260810 23:38:34.501950 1701 catalog_manager.cc:4841] TS d0f40688db8f4fb59a50f3f08710c6ef (127.30.33.3:35915): tablet 4ca45c8b3877492485a4f0a0f714c400 (table test-table [id=2717a45c84534ef6b7a418b13237980a]) successfully deleted
Aug 10 23:38:34 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405112, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:34.519948 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:34.476826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.30.33.1:43427)
0810 23:38:34.500587 (+ 23761us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:34.500592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:34.500615 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:38:34.500669 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:34.500672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:34.500741 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:34.500831 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:34.502500 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:34.503345 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:34.504684 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:34.505085 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:34.505194 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:34.513836 (+ 8642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:34.513857 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:34.513875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:34.513919 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:34.516908 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:34.517669 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:34.517677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:34.517680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:34.517828 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:34.518260 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:34.518263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:34.518266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:34.518669 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:34.518999 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:34.519329 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:38:34.519617 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23619,"thread_start_us":23496,"threads_started":1}
W20260810 23:38:34.520679 2097 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39208
W20260810 23:38:34.521435 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1852: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:34.521664 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 1852
I20260810 23:38:34.586706 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 2115
Aug 10 23:38:34 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405112, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:38:34.602766 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:34.587577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.30.33.2:36771)
0810 23:38:34.588136 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:34.588141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:34.588159 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:38:34.588216 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:34.588221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:34.588284 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:34.588373 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:34.589653 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:34.590425 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:34.591788 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:34.592243 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:34.592343 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:34.596715 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:34.596734 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:34.596758 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:34.596815 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:34.599975 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:34.600760 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:34.600769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:34.600772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:34.600935 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:34.601306 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:34.601309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:34.601311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:34.601713 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:34.601958 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:34.602224 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 23:38:34.602425 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":323,"threads_started":1}
W20260810 23:38:34.603515 2378 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46982
W20260810 23:38:34.604202 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 2115: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:34.604364 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 2115
I20260810 23:38:34.643138 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 2396
Aug 10 23:38:34 dist-test-slave-pww0 krb5kdc[1654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405112, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:38:34.660660 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:34.644065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.30.33.3:35915)
0810 23:38:34.644608 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:34.644613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:34.644630 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:38:34.644685 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:34.644689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:34.644772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:34.644860 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:34.646554 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:34.647329 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:34.649316 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:34.649775 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:34.649880 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:34.654342 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:34.654363 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:34.654380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:34.654423 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:34.657329 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:34.658386 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:34.658395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:34.658398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:34.658551 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:34.658982 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:34.658985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:34.658987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:34.659371 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:34.659664 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:34.660116 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 23:38:34.660339 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":265,"threads_started":1}
W20260810 23:38:34.661377 2643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42370
W20260810 23:38:34.662038 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 2396: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:34.662247 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 2396
I20260810 23:38:34.700834 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1674
I20260810 23:38:34.702087 1798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:34.823680 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 1674
2026-08-10T23:38:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9180 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:34 dist-test-slave-pww0 krb5kdc[2975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:34 dist-test-slave-pww0 krb5kdc[2975](info): set up 2 sockets
Aug 10 23:38:34 dist-test-slave-pww0 krb5kdc[2975](info): commencing operation
krb5kdc: starting...
W20260810 23:38:36.125460 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:36 dist-test-slave-pww0 krb5kdc[2975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:36.362936 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:39647
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:36.703017 3003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:36.703361 3003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:36.703538 3003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:36.711197 3003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:36.711344 3003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:36.711426 3003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:36.711530 3003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:36.711596 3003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:36.721994 3003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:36.723428 3003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:36.724892 3003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:36.737385 3011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:36.737308 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:36.737357 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:36.738245 3003 server_base.cc:1034] running on GCE node
I20260810 23:38:36.739084 3003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:36.740671 3003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:36.741895 3003 hybrid_clock.cc:648] HybridClock initialized: now 1786405116741845 us; error 46 us; skew 500 ppm
I20260810 23:38:36.742251 3003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:36 dist-test-slave-pww0 krb5kdc[2975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405116, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:36.745842 3003 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:36.747627 3003 webserver.cc:460] Webserver started at http://127.30.33.62:39663/ using document root <none> and password file <none>
I20260810 23:38:36.748368 3003 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:36.748492 3003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:36.748817 3003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:36.752035 3003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "737a1669538b48a38600e2c986886d80"
format_stamp: "Formatted at 2026-08-10 23:38:36 on dist-test-slave-pww0"
server_key: "199767585e17d0b3ab9cb9f15696f17a"
server_key_iv: "ea729564c4d94eb75b8340931f32b7ad"
server_key_version: "encryptionkey@0"
I20260810 23:38:36.752848 3003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "737a1669538b48a38600e2c986886d80"
format_stamp: "Formatted at 2026-08-10 23:38:36 on dist-test-slave-pww0"
server_key: "199767585e17d0b3ab9cb9f15696f17a"
server_key_iv: "ea729564c4d94eb75b8340931f32b7ad"
server_key_version: "encryptionkey@0"
I20260810 23:38:36.758754 3003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260810 23:38:36.763020 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:36.764750 3003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:36.764966 3003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "737a1669538b48a38600e2c986886d80"
format_stamp: "Formatted at 2026-08-10 23:38:36 on dist-test-slave-pww0"
server_key: "199767585e17d0b3ab9cb9f15696f17a"
server_key_iv: "ea729564c4d94eb75b8340931f32b7ad"
server_key_version: "encryptionkey@0"
I20260810 23:38:36.765208 3003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:36.785205 3003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:36.790091 3003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:36.790408 3003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:36.809484 3003 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:39647
I20260810 23:38:36.809533 3111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:39647 every 8 connection(s)
I20260810 23:38:36.811038 3003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:36.815037 3112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:36.816807 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3003
I20260810 23:38:36.817018 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:36.817658 30852 external_mini_cluster.cc:1428] Setting key 33bd4d72743dfa9981b693db7cbcdb50
I20260810 23:38:36.830837 3112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80: Bootstrap starting.
Aug 10 23:38:36 dist-test-slave-pww0 krb5kdc[2975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:36.834326 3112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:36.835847 3112 log.cc:824] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:36.839519 3112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80: No bootstrap required, opened a new log
I20260810 23:38:36.843032 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:36.819817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.30.33.62:39647)
0810 23:38:36.820755 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:36.820779 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:36.820837 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:38:36.822617 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:36.822651 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:36.823254 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:36.823777 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:36.825199 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:36.827265 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:36.828605 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:36.829748 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:36.829945 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:36.834063 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:36.834076 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:36.834098 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:36.834159 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:36.838022 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:36.838802 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:36.838813 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:36.838823 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:36.839052 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:36.839408 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:36.839413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:36.839415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:36.840262 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:36.840590 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:36.840989 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 23:38:36.841476 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":401,"threads_started":1}
I20260810 23:38:36.849013 3112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a1669538b48a38600e2c986886d80" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39647 } }
I20260810 23:38:36.849469 3112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:36.849571 3112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737a1669538b48a38600e2c986886d80, State: Initialized, Role: FOLLOWER
I20260810 23:38:36.850180 3112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a1669538b48a38600e2c986886d80" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39647 } }
I20260810 23:38:36.850441 3112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:36.850555 3112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:36.850713 3112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:36.853417 3112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a1669538b48a38600e2c986886d80" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39647 } }
I20260810 23:38:36.854043 3112 leader_election.cc:302] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 737a1669538b48a38600e2c986886d80; no voters:
I20260810 23:38:36.854842 3112 leader_election.cc:288] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:36.854983 3119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:36.856438 3119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [term 1 LEADER]: Becoming Leader. State: Replica: 737a1669538b48a38600e2c986886d80, State: Running, Role: LEADER
I20260810 23:38:36.857007 3119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a1669538b48a38600e2c986886d80" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39647 } }
I20260810 23:38:36.857285 3112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:36.859910 3123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737a1669538b48a38600e2c986886d80. Latest consensus state: current_term: 1 leader_uuid: "737a1669538b48a38600e2c986886d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a1669538b48a38600e2c986886d80" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39647 } } }
I20260810 23:38:36.860329 3123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:36.860875 3122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737a1669538b48a38600e2c986886d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737a1669538b48a38600e2c986886d80" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39647 } } }
I20260810 23:38:36.861181 3122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:36.867795 3130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:36.872704 3130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:36.881307 3130 catalog_manager.cc:1347] Generated new cluster ID: eaf704260bf446a188ab940e3188ef82
I20260810 23:38:36.881441 3130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:36.918499 3130 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:36.920009 3130 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:36.928853 3130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737a1669538b48a38600e2c986886d80: Generated new TSK 0
I20260810 23:38:36.929853 3130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:36.995440 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39647
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:37.347740 3163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:37.348057 3163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:37.348186 3163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:37.355808 3163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:37.355940 3163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:37.356058 3163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:37.366518 3163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:37.367935 3163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:37.369474 3163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:37.383384 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:37.383378 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:37.383504 3178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:37.384513 3163 server_base.cc:1034] running on GCE node
I20260810 23:38:37.385222 3163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:37.386227 3163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:37.387441 3163 hybrid_clock.cc:648] HybridClock initialized: now 1786405117387370 us; error 61 us; skew 500 ppm
I20260810 23:38:37.387861 3163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:37 dist-test-slave-pww0 krb5kdc[2975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:37.391199 3163 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:37.392894 3163 webserver.cc:460] Webserver started at http://127.30.33.1:34755/ using document root <none> and password file <none>
I20260810 23:38:37.393642 3163 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:37.393785 3163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:37.394104 3163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:37.397341 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "3ca86478791947c7a9784132ed9e4ba3"
format_stamp: "Formatted at 2026-08-10 23:38:37 on dist-test-slave-pww0"
server_key: "bd5444cb291d652833a42f3d78133315"
server_key_iv: "3c398ba1b920f85531476698b8652b52"
server_key_version: "encryptionkey@0"
I20260810 23:38:37.398221 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "3ca86478791947c7a9784132ed9e4ba3"
format_stamp: "Formatted at 2026-08-10 23:38:37 on dist-test-slave-pww0"
server_key: "bd5444cb291d652833a42f3d78133315"
server_key_iv: "3c398ba1b920f85531476698b8652b52"
server_key_version: "encryptionkey@0"
I20260810 23:38:37.404162 3163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 23:38:37.408207 3190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:37.409868 3163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:37.410056 3163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "3ca86478791947c7a9784132ed9e4ba3"
format_stamp: "Formatted at 2026-08-10 23:38:37 on dist-test-slave-pww0"
server_key: "bd5444cb291d652833a42f3d78133315"
server_key_iv: "3c398ba1b920f85531476698b8652b52"
server_key_version: "encryptionkey@0"
I20260810 23:38:37.410303 3163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:37.427402 3163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:37.431957 3163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:37.432247 3163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:37.433260 3163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:37.434684 3163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:37.434768 3163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:37.434890 3163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:37.434947 3163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:37.467840 3163 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:42141
I20260810 23:38:37.467895 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:42141 every 8 connection(s)
I20260810 23:38:37.469246 3163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:37.470369 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3163
I20260810 23:38:37.470603 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:37.471254 30852 external_mini_cluster.cc:1428] Setting key 977e6ee103374f02198e05175239193f
Aug 10 23:38:37 dist-test-slave-pww0 krb5kdc[2975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:37.491722 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:37.472187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:57359 (local address 127.30.33.62:39647)
0810 23:38:37.472657 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:37.472661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:37.474027 (+ 1366us) server_negotiation.cc:407] Connection header received
0810 23:38:37.476076 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:37.476082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:37.476187 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:37.476304 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:37.477992 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:37.478748 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:37.480517 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:37.480947 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:37.481076 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:37.485399 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:37.485405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:37.485408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:37.485448 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:37.488169 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:37.489077 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:37.489083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:37.489085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:37.489236 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:37.489739 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:37.489743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:37.489745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:37.490040 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:37.490202 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:37.491034 (+ 832us) server_negotiation.cc:299] Negotiation successful
0810 23:38:37.491310 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260810 23:38:37.492795 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:37.472862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39647 (local address 127.30.33.1:57359)
0810 23:38:37.473829 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:37.473888 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:37.475773 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:37.476523 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:37.476533 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:37.477139 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:37.477785 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:37.477806 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:37.478945 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:37.478949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:37.480336 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:37.480352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:37.481213 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:37.481220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:37.481510 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:37.482354 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:37.482387 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:37.485115 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:37.488378 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:37.488389 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:37.488405 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:37.488922 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:37.489364 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:37.489369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:37.489373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:37.489606 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:37.490182 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:37.490191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:37.490607 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 23:38:37.490881 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 23:38:37.491541 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":399,"threads_started":1}
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
I20260810 23:38:37.494544 3395 heartbeater.cc:347] Connected to a master server at 127.30.33.62:39647
I20260810 23:38:37.494879 3395 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:37.495756 3395 heartbeater.cc:511] Master 127.30.33.62:39647 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
I20260810 23:38:37.498178 3055 ts_manager.cc:194] Registered new tserver with Master: 3ca86478791947c7a9784132ed9e4ba3 (127.30.33.1:42141)
I20260810 23:38:37.499796 3055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:57359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:37.527853 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39647
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:37.869973 3404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:37.870286 3404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:37.870395 3404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:37.877774 3404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:37.877897 3404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:37.878023 3404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:37.888018 3404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:37.889367 3404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:37.890743 3404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:37.904134 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:37.904174 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:37.904139 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:37.905087 3404 server_base.cc:1034] running on GCE node
I20260810 23:38:37.905787 3404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:37.906828 3404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:37.908089 3404 hybrid_clock.cc:648] HybridClock initialized: now 1786405117908028 us; error 51 us; skew 500 ppm
I20260810 23:38:37.908464 3404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:37 dist-test-slave-pww0 krb5kdc[2975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:37.911952 3404 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:37.913488 3404 webserver.cc:460] Webserver started at http://127.30.33.2:33565/ using document root <none> and password file <none>
I20260810 23:38:37.914284 3404 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:37.914417 3404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:37.914743 3404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:37.917965 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "c52b16816f814db6843465746b44b3f2"
format_stamp: "Formatted at 2026-08-10 23:38:37 on dist-test-slave-pww0"
server_key: "998c51974391867957a24cf3ee0c5edb"
server_key_iv: "2769b5c7ead8008540bd8d07045484fa"
server_key_version: "encryptionkey@0"
I20260810 23:38:37.918802 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "c52b16816f814db6843465746b44b3f2"
format_stamp: "Formatted at 2026-08-10 23:38:37 on dist-test-slave-pww0"
server_key: "998c51974391867957a24cf3ee0c5edb"
server_key_iv: "2769b5c7ead8008540bd8d07045484fa"
server_key_version: "encryptionkey@0"
I20260810 23:38:37.924312 3404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:38:37.928001 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:37.929591 3404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:38:37.929780 3404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "c52b16816f814db6843465746b44b3f2"
format_stamp: "Formatted at 2026-08-10 23:38:37 on dist-test-slave-pww0"
server_key: "998c51974391867957a24cf3ee0c5edb"
server_key_iv: "2769b5c7ead8008540bd8d07045484fa"
server_key_version: "encryptionkey@0"
I20260810 23:38:37.930008 3404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:37.962615 3404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:37.967191 3404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:37.967535 3404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:37.968649 3404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:37.970320 3404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:37.970407 3404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:37.970542 3404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:37.970597 3404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:38.004436 3404 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:38521
I20260810 23:38:38.004511 3625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:38521 every 8 connection(s)
I20260810 23:38:38.005934 3404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:38.012415 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3404
I20260810 23:38:38.012625 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:38.013172 30852 external_mini_cluster.cc:1428] Setting key b3a67bbd69bbac537d8866d9c42674f1
Aug 10 23:38:38 dist-test-slave-pww0 krb5kdc[2975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:38.030046 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:38.008705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:43553 (local address 127.30.33.62:39647)
0810 23:38:38.009130 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:38.009134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:38.010288 (+ 1154us) server_negotiation.cc:407] Connection header received
0810 23:38:38.012316 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:38.012321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:38.012409 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:38.012512 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:38.014652 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.015555 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.017934 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.018396 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.018499 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:38.023226 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:38.023232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:38.023237 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:38.023282 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:38.026428 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:38.027337 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:38.027343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:38.027346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:38.027546 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:38.028038 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:38.028043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:38.028046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:38.028432 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:38.028600 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:38.029427 (+ 827us) server_negotiation.cc:299] Negotiation successful
0810 23:38:38.029677 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":208,"threads_started":1}
I20260810 23:38:38.030964 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:38.009316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39647 (local address 127.30.33.2:43553)
0810 23:38:38.010096 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:38.010143 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:38.011970 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:38.012840 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:38.012853 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:38.013690 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:38.014474 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:38.014502 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.015744 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.015749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:38.017770 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:38.017790 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.018649 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.018656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:38.019028 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:38.019950 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:38.019990 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:38.022879 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:38.026645 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:38.026654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:38.026670 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:38.027186 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:38.027687 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:38.027691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:38.027694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:38.027917 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:38.028614 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:38.028622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:38.029026 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 23:38:38.029290 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 23:38:38.029743 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":321,"threads_started":1}
I20260810 23:38:38.032509 3626 heartbeater.cc:347] Connected to a master server at 127.30.33.62:39647
I20260810 23:38:38.032829 3626 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:38.033656 3626 heartbeater.cc:511] Master 127.30.33.62:39647 requested a full tablet report, sending...
I20260810 23:38:38.035445 3055 ts_manager.cc:194] Registered new tserver with Master: c52b16816f814db6843465746b44b3f2 (127.30.33.2:38521)
I20260810 23:38:38.036693 3055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:43553
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:38.075500 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39647
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:38.430660 3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:38.430974 3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:38.431078 3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:38.439023 3636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:38.439162 3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:38.439291 3636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:38.449875 3636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:38.451284 3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:38.452708 3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:38.467391 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:38.467299 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:38.467707 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:38.469512 3636 server_base.cc:1034] running on GCE node
I20260810 23:38:38.470376 3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:38.472302 3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:38.473652 3636 hybrid_clock.cc:648] HybridClock initialized: now 1786405118473574 us; error 69 us; skew 500 ppm
I20260810 23:38:38.474072 3636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:38 dist-test-slave-pww0 krb5kdc[2975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405118, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:38.477849 3636 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:38.479606 3636 webserver.cc:460] Webserver started at http://127.30.33.3:37715/ using document root <none> and password file <none>
I20260810 23:38:38.480460 3636 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:38.480597 3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:38.480926 3636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:38.484292 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "1fb8eaf72de24b799afa209108e09888"
format_stamp: "Formatted at 2026-08-10 23:38:38 on dist-test-slave-pww0"
server_key: "51572f6af372b065c9cde68931aa64ce"
server_key_iv: "e9c58b039b4d54c70fe4df6502c84bc2"
server_key_version: "encryptionkey@0"
I20260810 23:38:38.485216 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "1fb8eaf72de24b799afa209108e09888"
format_stamp: "Formatted at 2026-08-10 23:38:38 on dist-test-slave-pww0"
server_key: "51572f6af372b065c9cde68931aa64ce"
server_key_iv: "e9c58b039b4d54c70fe4df6502c84bc2"
server_key_version: "encryptionkey@0"
I20260810 23:38:38.491153 3636 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:38:38.495488 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:38.497191 3636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:38:38.497401 3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "1fb8eaf72de24b799afa209108e09888"
format_stamp: "Formatted at 2026-08-10 23:38:38 on dist-test-slave-pww0"
server_key: "51572f6af372b065c9cde68931aa64ce"
server_key_iv: "e9c58b039b4d54c70fe4df6502c84bc2"
server_key_version: "encryptionkey@0"
I20260810 23:38:38.497635 3636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:38.503901 3395 heartbeater.cc:503] Master 127.30.33.62:39647 was elected leader, sending a full tablet report...
I20260810 23:38:38.511950 3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:38.516485 3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:38.516779 3636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:38.517815 3636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:38.519289 3636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:38.519362 3636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:38.519502 3636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:38.519567 3636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 23:38:38.550242 3636 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:38895
I20260810 23:38:38.550287 3854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:38895 every 8 connection(s)
I20260810 23:38:38.551640 3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:38.555367 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3636
I20260810 23:38:38.555605 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:38.556143 30852 external_mini_cluster.cc:1428] Setting key 7b7d0540d9589a4fe3e7cca31b804ee4
Aug 10 23:38:38 dist-test-slave-pww0 krb5kdc[2975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405118, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:38.572183 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:38.554275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:57491 (local address 127.30.33.62:39647)
0810 23:38:38.554784 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:38.554791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:38.555845 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 23:38:38.557856 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:38.557861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:38.557969 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:38.558058 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:38.559849 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.560637 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.562161 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.562558 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.562648 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:38.566169 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:38.566174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:38.566177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:38.566213 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:38.568773 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:38.569523 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:38.569528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:38.569531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:38.569715 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:38.570290 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:38.570295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:38.570299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:38.570616 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:38.570778 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:38.571542 (+ 764us) server_negotiation.cc:299] Negotiation successful
0810 23:38:38.571798 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":236,"threads_started":1}
I20260810 23:38:38.573172 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:38.554817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39647 (local address 127.30.33.3:57491)
0810 23:38:38.555687 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:38.555738 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:38.557564 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:38.558336 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:38.558347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:38.558932 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:38.559694 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:38.559716 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.560767 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.560771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:38.562025 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:38.562037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:38.562750 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:38.562755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:38.562959 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:38.563618 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:38.563640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:38.565944 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:38.568931 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:38.568939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:38.568953 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:38.569404 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:38.569876 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:38.569880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:38.569883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:38.570146 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:38.570753 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:38.570761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:38.571124 (+ 363us) client_negotiation.cc:769] Sending connection context
0810 23:38:38.571380 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 23:38:38.571997 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":335,"threads_started":1}
I20260810 23:38:38.574587 3857 heartbeater.cc:347] Connected to a master server at 127.30.33.62:39647
I20260810 23:38:38.574922 3857 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:38.575728 3857 heartbeater.cc:511] Master 127.30.33.62:39647 requested a full tablet report, sending...
I20260810 23:38:38.577216 3055 ts_manager.cc:194] Registered new tserver with Master: 1fb8eaf72de24b799afa209108e09888 (127.30.33.3:38895)
I20260810 23:38:38.578065 3055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:57491
I20260810 23:38:38.584117 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:38:38.625041 3859 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:38.622346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.30.33.62:39647)
0810 23:38:38.622603 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:38.622611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:38.622957 (+ 346us) server_negotiation.cc:407] Connection header received
0810 23:38:38.623153 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:38.623158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:38.623233 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:38.623321 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:38.624765 (+ 1444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60328: BlockingRecv error: recv got EOF from 127.0.0.1:60328 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:38:38.628443 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3163
W20260810 23:38:38.631414 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3163: Not authorized: Client connection negotiation failed: client connection to 127.30.33.1:42141: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:38:38.631623 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3163
I20260810 23:38:38.687620 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3404
W20260810 23:38:38.690361 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3404: Not authorized: Client connection negotiation failed: client connection to 127.30.33.2:38521: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:38:38.690558 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3404
I20260810 23:38:38.715058 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3636
W20260810 23:38:38.718004 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3636: Not authorized: Client connection negotiation failed: client connection to 127.30.33.3:38895: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:38:38.718158 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3636
I20260810 23:38:38.741272 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3003
I20260810 23:38:38.742530 3102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:38.849146 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3003
2026-08-10T23:38:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4019 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:38 dist-test-slave-pww0 krb5kdc[3890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:38 dist-test-slave-pww0 krb5kdc[3890](info): set up 2 sockets
Aug 10 23:38:38 dist-test-slave-pww0 krb5kdc[3890](info): commencing operation
krb5kdc: starting...
W20260810 23:38:40.116489 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:40 dist-test-slave-pww0 krb5kdc[3890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:40.369306 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:34013
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:34013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:40.711195 3913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:40.711551 3913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:40.711688 3913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:40.719381 3913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:40.719564 3913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:40.719653 3913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:40.719743 3913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:40.719810 3913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:40.730456 3913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:34013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:34013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:40.732043 3913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:40.733546 3913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:40.746045 3929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:40.746138 3927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:40.746119 3925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:40.747035 3913 server_base.cc:1034] running on GCE node
I20260810 23:38:40.748008 3913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:40.749636 3913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:40.750864 3913 hybrid_clock.cc:648] HybridClock initialized: now 1786405120750815 us; error 43 us; skew 500 ppm
I20260810 23:38:40.751259 3913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:40 dist-test-slave-pww0 krb5kdc[3890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405120, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:40.755069 3913 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:40.756932 3913 webserver.cc:460] Webserver started at http://127.30.33.62:46195/ using document root <none> and password file <none>
I20260810 23:38:40.757747 3913 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:40.757884 3913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:40.758220 3913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:40.761593 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "ef00e819f62743c5a6153f0b8fc59935"
format_stamp: "Formatted at 2026-08-10 23:38:40 on dist-test-slave-pww0"
server_key: "512aa6c5a9e487a55a4a3a90ad8a4d0f"
server_key_iv: "52ee823ca8f8cc3a90e1a4d68e4a8074"
server_key_version: "encryptionkey@0"
I20260810 23:38:40.762514 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "ef00e819f62743c5a6153f0b8fc59935"
format_stamp: "Formatted at 2026-08-10 23:38:40 on dist-test-slave-pww0"
server_key: "512aa6c5a9e487a55a4a3a90ad8a4d0f"
server_key_iv: "52ee823ca8f8cc3a90e1a4d68e4a8074"
server_key_version: "encryptionkey@0"
I20260810 23:38:40.768249 3913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:38:40.772014 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:40.773449 3913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:38:40.773634 3913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "ef00e819f62743c5a6153f0b8fc59935"
format_stamp: "Formatted at 2026-08-10 23:38:40 on dist-test-slave-pww0"
server_key: "512aa6c5a9e487a55a4a3a90ad8a4d0f"
server_key_iv: "52ee823ca8f8cc3a90e1a4d68e4a8074"
server_key_version: "encryptionkey@0"
I20260810 23:38:40.773833 3913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:40.798049 3913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:40.802505 3913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:40.802811 3913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:40.823551 3913 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:34013
I20260810 23:38:40.823553 4029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:34013 every 8 connection(s)
I20260810 23:38:40.825049 3913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:40.829003 4033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:40.833828 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3913
I20260810 23:38:40.834048 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:40.834676 30852 external_mini_cluster.cc:1428] Setting key 7b008cef83cead8f706010ba87a06725
I20260810 23:38:40.840514 4033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935: Bootstrap starting.
I20260810 23:38:40.844395 4033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:40.845651 4033 log.cc:824] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:40.849149 4033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935: No bootstrap required, opened a new log
Aug 10 23:38:40 dist-test-slave-pww0 krb5kdc[3890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:40.857560 4033 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00e819f62743c5a6153f0b8fc59935" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34013 } }
I20260810 23:38:40.858018 4033 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:40.858132 4033 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef00e819f62743c5a6153f0b8fc59935, State: Initialized, Role: FOLLOWER
I20260810 23:38:40.858894 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:40.836573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.30.33.62:34013)
0810 23:38:40.837448 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:40.837462 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:40.837509 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:38:40.839332 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:40.839358 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:40.839838 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:40.840185 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:40.842134 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:40.843533 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:40.845211 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:40.845968 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:40.846108 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:40.850701 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:40.850709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:40.850732 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:40.850774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:40.854150 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:40.854949 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:40.854956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:40.854964 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:40.855177 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:40.855627 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:40.855631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:40.855633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:40.856291 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:40.856618 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:40.857141 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 23:38:40.857623 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":398,"threads_started":1}
I20260810 23:38:40.858798 4033 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00e819f62743c5a6153f0b8fc59935" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34013 } }
I20260810 23:38:40.859061 4033 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:40.859215 4033 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:40.859421 4033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:40.862424 4033 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00e819f62743c5a6153f0b8fc59935" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34013 } }
I20260810 23:38:40.863044 4033 leader_election.cc:302] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef00e819f62743c5a6153f0b8fc59935; no voters:
I20260810 23:38:40.863323 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:38:40.863754 4033 leader_election.cc:288] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:40.863886 4051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:40.866423 4033 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:40.867760 4051 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [term 1 LEADER]: Becoming Leader. State: Replica: ef00e819f62743c5a6153f0b8fc59935, State: Running, Role: LEADER
I20260810 23:38:40.868585 4051 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00e819f62743c5a6153f0b8fc59935" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34013 } }
I20260810 23:38:40.872092 4058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef00e819f62743c5a6153f0b8fc59935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00e819f62743c5a6153f0b8fc59935" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34013 } } }
I20260810 23:38:40.872479 4058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:40.872751 4057 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef00e819f62743c5a6153f0b8fc59935. Latest consensus state: current_term: 1 leader_uuid: "ef00e819f62743c5a6153f0b8fc59935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00e819f62743c5a6153f0b8fc59935" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34013 } } }
I20260810 23:38:40.873011 4057 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:40.874248 4079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:40.891515 4079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:40.902021 4079 catalog_manager.cc:1347] Generated new cluster ID: 5a898efe2967467cb07304ff1d06e79c
I20260810 23:38:40.902166 4079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:40.915642 4079 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:40.917312 4079 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:40.925076 4079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef00e819f62743c5a6153f0b8fc59935: Generated new TSK 0
I20260810 23:38:40.926288 4079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:38:40.930526 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:40.922200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.30.33.62:34013)
0810 23:38:40.922434 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:40.922441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:40.923106 (+ 665us) server_negotiation.cc:407] Connection header received
0810 23:38:40.923326 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:40.923331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:40.923426 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:40.923579 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:40.924842 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:40.925669 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:40.927119 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:40.927628 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:40.927759 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:40.928952 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:40.928957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:40.928961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:40.929660 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:40.929740 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:40.929778 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:40.929881 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:40.930047 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:38:40.930163 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 23:38:40.938805 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3913
I20260810 23:38:40.940059 4020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:41.033901 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3913
2026-08-10T23:38:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2176 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:41 dist-test-slave-pww0 krb5kdc[4108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:41 dist-test-slave-pww0 krb5kdc[4108](info): set up 2 sockets
Aug 10 23:38:41 dist-test-slave-pww0 krb5kdc[4108](info): commencing operation
krb5kdc: starting...
W20260810 23:38:42.312629 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:42 dist-test-slave-pww0 krb5kdc[4108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:42.558717 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:38299
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:38299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:42.899194 4125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:42.899559 4125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:42.899719 4125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:42.907078 4125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:42.907199 4125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:42.907261 4125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:42.907346 4125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:42.907403 4125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:42.917649 4125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:38299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:38299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:42.919094 4125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:42.920469 4125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:42.932134 4131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:42.932163 4134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:42.932215 4137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:42.933283 4125 server_base.cc:1034] running on GCE node
I20260810 23:38:42.934103 4125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:42.935566 4125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:42.936777 4125 hybrid_clock.cc:648] HybridClock initialized: now 1786405122936726 us; error 37 us; skew 500 ppm
I20260810 23:38:42.937103 4125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:42 dist-test-slave-pww0 krb5kdc[4108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405122, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:42.940443 4125 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:42.942194 4125 webserver.cc:460] Webserver started at http://127.30.33.62:39113/ using document root <none> and password file <none>
I20260810 23:38:42.942958 4125 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:42.943078 4125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:42.943408 4125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:42.946722 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b"
format_stamp: "Formatted at 2026-08-10 23:38:42 on dist-test-slave-pww0"
server_key: "debf72a8ebdb02e9780747250ae31914"
server_key_iv: "c2f3ebdde323d70606e911c69d40b9d7"
server_key_version: "encryptionkey@0"
I20260810 23:38:42.947643 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b"
format_stamp: "Formatted at 2026-08-10 23:38:42 on dist-test-slave-pww0"
server_key: "debf72a8ebdb02e9780747250ae31914"
server_key_iv: "c2f3ebdde323d70606e911c69d40b9d7"
server_key_version: "encryptionkey@0"
I20260810 23:38:42.953413 4125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 23:38:42.957235 4147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:42.958671 4125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:38:42.958837 4125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b"
format_stamp: "Formatted at 2026-08-10 23:38:42 on dist-test-slave-pww0"
server_key: "debf72a8ebdb02e9780747250ae31914"
server_key_iv: "c2f3ebdde323d70606e911c69d40b9d7"
server_key_version: "encryptionkey@0"
I20260810 23:38:42.959026 4125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:42.974233 4125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:42.978740 4125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:42.979040 4125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:42.997234 4125 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:38299
I20260810 23:38:42.997258 4249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:38299 every 8 connection(s)
I20260810 23:38:42.998733 4125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:43.002771 4250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:43.003149 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4125
I20260810 23:38:43.003384 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:43.004297 30852 external_mini_cluster.cc:1428] Setting key f4955882c1f128c3522d6d0f20c9333e
I20260810 23:38:43.018985 4250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b: Bootstrap starting.
Aug 10 23:38:43 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:43.022368 4250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:43.023821 4250 log.cc:824] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:43.028057 4250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b: No bootstrap required, opened a new log
I20260810 23:38:43.031747 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:43.006670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.30.33.62:38299)
0810 23:38:43.007563 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:43.007579 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:43.007638 (+ 59us) server_negotiation.cc:407] Connection header received
0810 23:38:43.010659 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:43.010707 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:43.011342 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:43.011933 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:43.013502 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:43.015581 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:43.016888 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:43.017963 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:43.018149 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:43.022431 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:43.022444 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:43.022466 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:43.022524 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:43.026736 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:43.027571 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:43.027580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:43.027589 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:43.027828 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:43.028229 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:43.028234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:43.028237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:43.029032 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:43.029354 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:43.029724 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 23:38:43.030218 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":348,"threads_started":1}
I20260810 23:38:43.037623 4250 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38299 } }
I20260810 23:38:43.038043 4250 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:43.038146 4250 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb6ec2980ed4e348f9e2a73e51c8e4b, State: Initialized, Role: FOLLOWER
I20260810 23:38:43.038780 4250 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38299 } }
I20260810 23:38:43.039004 4250 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:43.039109 4250 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:43.039264 4250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:43.041607 4250 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38299 } }
I20260810 23:38:43.042119 4250 leader_election.cc:302] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbb6ec2980ed4e348f9e2a73e51c8e4b; no voters:
I20260810 23:38:43.042765 4250 leader_election.cc:288] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:43.042902 4263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:43.043581 4263 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [term 1 LEADER]: Becoming Leader. State: Replica: fbb6ec2980ed4e348f9e2a73e51c8e4b, State: Running, Role: LEADER
I20260810 23:38:43.044195 4263 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38299 } }
I20260810 23:38:43.045166 4250 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:43.046909 4268 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38299 } } }
I20260810 23:38:43.047008 4265 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb6ec2980ed4e348f9e2a73e51c8e4b. Latest consensus state: current_term: 1 leader_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb6ec2980ed4e348f9e2a73e51c8e4b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38299 } } }
I20260810 23:38:43.047325 4268 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:43.047371 4265 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:43.048182 4272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:43.053681 4272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:43.063827 4272 catalog_manager.cc:1347] Generated new cluster ID: c75473c95a6746868ded3a6b0733f98d
I20260810 23:38:43.063962 4272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:43.078985 4272 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:43.080441 4272 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:43.089182 4272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbb6ec2980ed4e348f9e2a73e51c8e4b: Generated new TSK 0
I20260810 23:38:43.090132 4272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:43.151816 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:38299
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:43.514099 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:43.514429 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:43.514530 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:43.522251 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:43.522382 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:43.522502 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:43.533439 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:38299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:43.534791 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:43.536207 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:43.550081 4311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:43.550114 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:43.550379 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:43.550954 4300 server_base.cc:1034] running on GCE node
I20260810 23:38:43.551630 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:43.552616 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:43.553802 4300 hybrid_clock.cc:648] HybridClock initialized: now 1786405123553737 us; error 58 us; skew 500 ppm
I20260810 23:38:43.554175 4300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:43 dist-test-slave-pww0 krb5kdc[4108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405123, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:43.557652 4300 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:43.559140 4300 webserver.cc:460] Webserver started at http://127.30.33.1:43105/ using document root <none> and password file <none>
I20260810 23:38:43.559904 4300 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:43.560014 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:43.560298 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:43.563434 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "f5f23e2b853b4c7681d3ae1fb7d8122e"
format_stamp: "Formatted at 2026-08-10 23:38:43 on dist-test-slave-pww0"
server_key: "913d9db60fcc4560ebbc70bf6d9de6f9"
server_key_iv: "f4632955e57a4b68ee1da9b8de70a53b"
server_key_version: "encryptionkey@0"
I20260810 23:38:43.564309 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "f5f23e2b853b4c7681d3ae1fb7d8122e"
format_stamp: "Formatted at 2026-08-10 23:38:43 on dist-test-slave-pww0"
server_key: "913d9db60fcc4560ebbc70bf6d9de6f9"
server_key_iv: "f4632955e57a4b68ee1da9b8de70a53b"
server_key_version: "encryptionkey@0"
I20260810 23:38:43.569890 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:38:43.573832 4319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:43.575327 4300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:38:43.575529 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "f5f23e2b853b4c7681d3ae1fb7d8122e"
format_stamp: "Formatted at 2026-08-10 23:38:43 on dist-test-slave-pww0"
server_key: "913d9db60fcc4560ebbc70bf6d9de6f9"
server_key_iv: "f4632955e57a4b68ee1da9b8de70a53b"
server_key_version: "encryptionkey@0"
I20260810 23:38:43.575752 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:43.601307 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:43.605852 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:43.606156 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:43.607183 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:43.608718 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:43.608808 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:43.608938 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:43.608999 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:43.641149 4300 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:37141
I20260810 23:38:43.641201 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:37141 every 8 connection(s)
I20260810 23:38:43.642561 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:43.649396 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4300
I20260810 23:38:43.649595 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:43.650260 30852 external_mini_cluster.cc:1428] Setting key bb17b79c25e66f4ac1965a9547b7ccd3
Aug 10 23:38:43 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405123, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:43.665035 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:43.645344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:46467 (local address 127.30.33.62:38299)
0810 23:38:43.645832 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:43.645837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:43.646927 (+ 1090us) server_negotiation.cc:407] Connection header received
0810 23:38:43.648975 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:43.648981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:43.649086 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:43.649189 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:43.651040 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:43.651943 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:43.653813 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:43.654494 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:43.654624 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:43.658372 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:43.658380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:43.658385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:43.658429 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:43.661457 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:43.662498 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:43.662503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:43.662505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:43.662665 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:43.663148 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:43.663151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:43.663153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:43.663534 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:43.663667 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:43.664580 (+ 913us) server_negotiation.cc:299] Negotiation successful
0810 23:38:43.664771 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":233,"threads_started":1}
I20260810 23:38:43.666566 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:43.645934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:38299 (local address 127.30.33.1:46467)
0810 23:38:43.646752 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:43.646795 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:43.648658 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:43.649477 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:43.649487 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:43.650095 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:43.650796 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:43.650825 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:43.652175 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:43.652180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:43.653653 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:43.653668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:43.654715 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:43.654721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:43.654951 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:43.655676 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:43.655701 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:43.658097 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:43.661637 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:43.661648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:43.661670 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:43.662345 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:43.662779 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:43.662784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:43.662787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:43.663047 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:43.663668 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:43.663677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:43.664141 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 23:38:43.664476 (+ 335us) client_negotiation.cc:240] Negotiation successful
0810 23:38:43.665054 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":298,"threads_started":1}
I20260810 23:38:43.668229 4524 heartbeater.cc:347] Connected to a master server at 127.30.33.62:38299
I20260810 23:38:43.668551 4524 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:43.669327 4524 heartbeater.cc:511] Master 127.30.33.62:38299 requested a full tablet report, sending...
I20260810 23:38:43.671486 4182 ts_manager.cc:194] Registered new tserver with Master: f5f23e2b853b4c7681d3ae1fb7d8122e (127.30.33.1:37141)
I20260810 23:38:43.672984 4182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:46467
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:43.705070 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:38299
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:44.059320 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:44.059707 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:44.059864 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:44.067869 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:44.068019 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:44.068205 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:44.078486 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:38299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:44.080013 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:44.081566 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:44.096124 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:44.096143 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:44.096175 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:44.096987 4534 server_base.cc:1034] running on GCE node
I20260810 23:38:44.097685 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:44.098838 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:44.100097 4534 hybrid_clock.cc:648] HybridClock initialized: now 1786405124100027 us; error 50 us; skew 500 ppm
I20260810 23:38:44.100483 4534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:44.103977 4534 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:44.105706 4534 webserver.cc:460] Webserver started at http://127.30.33.2:44241/ using document root <none> and password file <none>
I20260810 23:38:44.106429 4534 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:44.106542 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:44.106844 4534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:44.110059 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "312cb4308fe548bcb615559592dd8320"
format_stamp: "Formatted at 2026-08-10 23:38:44 on dist-test-slave-pww0"
server_key: "e509aed98b423fb5a3e2ca3a475abfbb"
server_key_iv: "a152f4d4d03643f5e1ba751f9e22a336"
server_key_version: "encryptionkey@0"
I20260810 23:38:44.110915 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "312cb4308fe548bcb615559592dd8320"
format_stamp: "Formatted at 2026-08-10 23:38:44 on dist-test-slave-pww0"
server_key: "e509aed98b423fb5a3e2ca3a475abfbb"
server_key_iv: "a152f4d4d03643f5e1ba751f9e22a336"
server_key_version: "encryptionkey@0"
I20260810 23:38:44.116888 4534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260810 23:38:44.120929 4565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:44.122553 4534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:44.122743 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "312cb4308fe548bcb615559592dd8320"
format_stamp: "Formatted at 2026-08-10 23:38:44 on dist-test-slave-pww0"
server_key: "e509aed98b423fb5a3e2ca3a475abfbb"
server_key_iv: "a152f4d4d03643f5e1ba751f9e22a336"
server_key_version: "encryptionkey@0"
I20260810 23:38:44.122956 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:44.146108 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:44.150665 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:44.150960 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:44.152173 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:44.153798 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:44.153887 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:44.154004 4534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:44.154062 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:44.187301 4534 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:45609
I20260810 23:38:44.187355 4766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:45609 every 8 connection(s)
I20260810 23:38:44.188683 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:44.189841 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4534
I20260810 23:38:44.190047 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:44.190667 30852 external_mini_cluster.cc:1428] Setting key cf2384f3a168159f89c8e0106d709591
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:44.213903 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.192858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:37395 (local address 127.30.33.62:38299)
0810 23:38:44.193333 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:44.193339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:44.193387 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:38:44.196163 (+ 2776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:44.196168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:44.196256 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:44.196358 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:44.198729 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.199592 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.201852 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.202331 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.202439 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.206534 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:44.206541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:44.206545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:44.206592 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:44.210036 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.211159 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.211164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.211166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.211387 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.211967 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.211972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.211976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.212292 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:44.212430 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:44.213250 (+ 820us) server_negotiation.cc:299] Negotiation successful
0810 23:38:44.213508 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":2637,"threads_started":1}
I20260810 23:38:44.215029 4772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.192301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:38299 (local address 127.30.33.2:37395)
0810 23:38:44.193226 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:44.193295 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:44.195774 (+ 2479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:44.196673 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:44.196686 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:44.197500 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:44.198431 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:44.198476 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.199787 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.199793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:44.201658 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:44.201677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.202588 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.202595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:44.202897 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.203680 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:44.203705 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:44.206229 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:44.210265 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:44.210274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:44.210290 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:44.210992 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:44.211569 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:44.211576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:44.211580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:44.211846 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:44.212443 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:44.212452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:44.212853 (+ 401us) client_negotiation.cc:769] Sending connection context
0810 23:38:44.213131 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 23:38:44.213771 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":334,"threads_started":1}
I20260810 23:38:44.216648 4769 heartbeater.cc:347] Connected to a master server at 127.30.33.62:38299
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
I20260810 23:38:44.217060 4769 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:44.217969 4769 heartbeater.cc:511] Master 127.30.33.62:38299 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
I20260810 23:38:44.219712 4182 ts_manager.cc:194] Registered new tserver with Master: 312cb4308fe548bcb615559592dd8320 (127.30.33.2:45609)
I20260810 23:38:44.220610 4182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:37395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:44.248159 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:38299
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:44.611702 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:44.612047 4780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:44.612155 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:44.619951 4780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:44.620105 4780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:44.620265 4780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:44.630909 4780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:38299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:44.632418 4780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:44.633867 4780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:44.650295 4788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:44.650208 4791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:44.650241 4794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:44.650887 4780 server_base.cc:1034] running on GCE node
I20260810 23:38:44.651767 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:44.652878 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:44.654136 4780 hybrid_clock.cc:648] HybridClock initialized: now 1786405124654066 us; error 67 us; skew 500 ppm
I20260810 23:38:44.654505 4780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:44.658078 4780 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:44.659855 4780 webserver.cc:460] Webserver started at http://127.30.33.3:45259/ using document root <none> and password file <none>
I20260810 23:38:44.660641 4780 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:44.660786 4780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:44.661127 4780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:44.664727 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "eaceb5fa932d46a9ad455d5de34906d4"
format_stamp: "Formatted at 2026-08-10 23:38:44 on dist-test-slave-pww0"
server_key: "d48de9aa60dca503a20a790fd48aad52"
server_key_iv: "cd6432b5577707c8370976e2153acd5f"
server_key_version: "encryptionkey@0"
I20260810 23:38:44.665629 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "eaceb5fa932d46a9ad455d5de34906d4"
format_stamp: "Formatted at 2026-08-10 23:38:44 on dist-test-slave-pww0"
server_key: "d48de9aa60dca503a20a790fd48aad52"
server_key_iv: "cd6432b5577707c8370976e2153acd5f"
server_key_version: "encryptionkey@0"
I20260810 23:38:44.671701 4780 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:38:44.675578 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:44.676164 4524 heartbeater.cc:503] Master 127.30.33.62:38299 was elected leader, sending a full tablet report...
I20260810 23:38:44.677337 4780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:44.677536 4780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "eaceb5fa932d46a9ad455d5de34906d4"
format_stamp: "Formatted at 2026-08-10 23:38:44 on dist-test-slave-pww0"
server_key: "d48de9aa60dca503a20a790fd48aad52"
server_key_iv: "cd6432b5577707c8370976e2153acd5f"
server_key_version: "encryptionkey@0"
I20260810 23:38:44.677752 4780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:44.698796 4780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:44.703814 4780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:44.704145 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:44.705251 4780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:44.706856 4780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:44.706954 4780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:44.707101 4780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:44.707165 4780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:44.740298 4780 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:33637
I20260810 23:38:44.740348 5028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:33637 every 8 connection(s)
I20260810 23:38:44.741770 4780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:44.744354 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4780
I20260810 23:38:44.744539 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:44.745107 30852 external_mini_cluster.cc:1428] Setting key fea7c3804af68f2988205325fea08778
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:44.764127 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.744550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:45935 (local address 127.30.33.62:38299)
0810 23:38:44.745080 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:44.745085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:44.746522 (+ 1437us) server_negotiation.cc:407] Connection header received
0810 23:38:44.749093 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:44.749097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:44.749193 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:44.749292 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:44.751041 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.751956 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.753625 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.754090 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.754186 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.757811 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:44.757816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:44.757820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:44.757858 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:44.760843 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.761658 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.761662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.761665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.761810 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.762245 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.762249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.762251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.762555 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:44.762731 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:44.763496 (+ 765us) server_negotiation.cc:299] Negotiation successful
0810 23:38:44.763744 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":266,"threads_started":1}
I20260810 23:38:44.765157 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.745313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:38299 (local address 127.30.33.3:45935)
0810 23:38:44.746321 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:44.746383 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:44.748692 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:44.749612 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:44.749622 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:44.750204 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:44.750847 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:44.750867 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.752105 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.752109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:44.753497 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:44.753511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.754286 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.754291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:44.754498 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.755193 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:44.755224 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:44.757564 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:44.760992 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:44.761001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:44.761016 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:44.761511 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:44.761923 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:44.761927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:44.761929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:44.762128 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:44.762685 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:44.762701 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:44.763077 (+ 376us) client_negotiation.cc:769] Sending connection context
0810 23:38:44.763349 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 23:38:44.763984 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":386,"threads_started":1}
I20260810 23:38:44.766633 5029 heartbeater.cc:347] Connected to a master server at 127.30.33.62:38299
I20260810 23:38:44.766927 5029 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:44.767711 5029 heartbeater.cc:511] Master 127.30.33.62:38299 requested a full tablet report, sending...
I20260810 23:38:44.769297 4182 ts_manager.cc:194] Registered new tserver with Master: eaceb5fa932d46a9ad455d5de34906d4 (127.30.33.3:33637)
I20260810 23:38:44.770148 4182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:45935
I20260810 23:38:44.774075 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:44.834743 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.819313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.30.33.62:38299)
0810 23:38:44.819618 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:44.819625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:44.819922 (+ 297us) server_negotiation.cc:407] Connection header received
0810 23:38:44.820135 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:44.820140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:44.820229 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:44.820323 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:44.821804 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.822644 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.824052 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.824459 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.824566 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.829027 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:44.829032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:44.829036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:44.829078 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:44.831796 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.832609 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.832614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.832617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.832771 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.833189 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.833199 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.833201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.833637 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:44.833888 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:44.834227 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:38:44.834445 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 23:38:44.835358 4182 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49796
I20260810 23:38:44.836948 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4300
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:44.853318 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.838112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.30.33.1:37141)
0810 23:38:44.838715 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:44.838722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:44.838749 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:38:44.838807 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:44.838814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:44.839024 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:44.839171 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:44.840373 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.841438 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.842595 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.843158 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.843259 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.847418 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:44.847436 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:44.847452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:44.847537 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:44.850335 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.851074 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.851083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.851086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.851237 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.851733 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.851744 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.851746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.852160 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:44.852415 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:44.852683 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:38:44.852924 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
W20260810 23:38:44.854709 4516 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50794
W20260810 23:38:44.855942 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:44.856150 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4300
I20260810 23:38:44.883826 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4534
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:38:44.901775 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.884792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.30.33.2:45609)
0810 23:38:44.885357 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:44.885365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:44.885392 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:38:44.885459 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:44.885465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:44.885687 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:44.885837 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:44.887115 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.888264 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.889774 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.890398 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.890513 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.895185 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:44.895210 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:44.895229 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:44.895285 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:44.898557 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.899444 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.899487 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.899491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.899672 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.900058 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.900062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.900064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.900458 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:44.900741 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:44.901109 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 23:38:44.901375 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":284,"threads_started":1}
W20260810 23:38:44.903165 4762 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58818
W20260810 23:38:44.904445 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:44.904618 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4534
I20260810 23:38:44.929409 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4780
Aug 10 23:38:44 dist-test-slave-pww0 krb5kdc[4108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405124, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:38:44.948258 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:44.930598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.30.33.3:33637)
0810 23:38:44.931320 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:44.931328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:44.931504 (+ 176us) server_negotiation.cc:407] Connection header received
0810 23:38:44.931623 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:44.931633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:44.931925 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:44.932085 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:44.933367 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.934441 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.935781 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:44.936406 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:44.936508 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:44.941609 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:44.941626 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:44.941644 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:44.941692 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:44.945032 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.945976 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.945985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.945987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.946161 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:44.946605 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:44.946609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:44.946632 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:44.947063 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:44.947320 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:44.947694 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 23:38:44.947941 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":371,"threads_started":1}
W20260810 23:38:44.949648 5023 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45090
W20260810 23:38:44.950938 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4780: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:38:44.951154 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4780
I20260810 23:38:44.974046 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4125
I20260810 23:38:44.975412 4237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:45.083668 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4125
2026-08-10T23:38:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4066 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:45 dist-test-slave-pww0 krb5kdc[5059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:45 dist-test-slave-pww0 krb5kdc[5059](info): set up 2 sockets
Aug 10 23:38:45 dist-test-slave-pww0 krb5kdc[5059](info): commencing operation
krb5kdc: starting...
W20260810 23:38:46.370783 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:46 dist-test-slave-pww0 krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:46.615792 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40449
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:40449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:46.965690 5085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:46.966033 5085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:46.966181 5085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:46.973654 5085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:46.973798 5085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:46.973888 5085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:46.973968 5085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:46.974031 5085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:46.984476 5085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:40449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:46.985903 5085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:46.987282 5085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:47.000200 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:47.000248 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:47.000176 5098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:47.002295 5085 server_base.cc:1034] running on GCE node
I20260810 23:38:47.003275 5085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:47.005077 5085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:47.006343 5085 hybrid_clock.cc:648] HybridClock initialized: now 1786405127006273 us; error 59 us; skew 500 ppm
I20260810 23:38:47.006748 5085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:47 dist-test-slave-pww0 krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405127, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:47.010097 5085 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:47.011818 5085 webserver.cc:460] Webserver started at http://127.30.33.62:43567/ using document root <none> and password file <none>
I20260810 23:38:47.012578 5085 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:47.012708 5085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:47.013041 5085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:47.016357 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "53c362aad0574cceb808d74fcc19c592"
format_stamp: "Formatted at 2026-08-10 23:38:47 on dist-test-slave-pww0"
server_key: "61bb31d0c592da3baf63a6512523ee94"
server_key_iv: "a17af5ec0a797c40144e3c9825d07cec"
server_key_version: "encryptionkey@0"
I20260810 23:38:47.017269 5085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "53c362aad0574cceb808d74fcc19c592"
format_stamp: "Formatted at 2026-08-10 23:38:47 on dist-test-slave-pww0"
server_key: "61bb31d0c592da3baf63a6512523ee94"
server_key_iv: "a17af5ec0a797c40144e3c9825d07cec"
server_key_version: "encryptionkey@0"
I20260810 23:38:47.023808 5085 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 23:38:47.028502 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:47.030067 5085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:38:47.030292 5085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "53c362aad0574cceb808d74fcc19c592"
format_stamp: "Formatted at 2026-08-10 23:38:47 on dist-test-slave-pww0"
server_key: "61bb31d0c592da3baf63a6512523ee94"
server_key_iv: "a17af5ec0a797c40144e3c9825d07cec"
server_key_version: "encryptionkey@0"
I20260810 23:38:47.030525 5085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:47.054777 5085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:47.059579 5085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:47.059873 5085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:47.078734 5085 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:40449
I20260810 23:38:47.078761 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:40449 every 8 connection(s)
I20260810 23:38:47.080255 5085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:47.081204 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5085
I20260810 23:38:47.081451 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:47.082232 30852 external_mini_cluster.cc:1428] Setting key 4b911bfaefb8f01185498c7b0f09c4be
I20260810 23:38:47.085311 5201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:38:47 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:47.099359 5201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592: Bootstrap starting.
I20260810 23:38:47.103665 5201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:47.104895 5201 log.cc:824] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:47.108075 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:47.084413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.30.33.62:40449)
0810 23:38:47.085258 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:47.085271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:47.085319 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:38:47.087051 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:47.087077 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:47.087538 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:47.087913 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:47.089837 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:47.091293 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:47.093136 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:47.093879 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:47.094018 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:47.098868 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:47.098881 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:47.098905 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:47.098962 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:47.102982 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:47.103924 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:47.103932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:47.103940 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:47.104150 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:47.104652 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:47.104656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:47.104659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:47.105332 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:47.105693 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:47.106150 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 23:38:47.106655 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":393,"threads_started":1}
I20260810 23:38:47.108544 5201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592: No bootstrap required, opened a new log
I20260810 23:38:47.116839 5201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c362aad0574cceb808d74fcc19c592" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40449 } }
I20260810 23:38:47.117272 5201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:47.117394 5201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c362aad0574cceb808d74fcc19c592, State: Initialized, Role: FOLLOWER
I20260810 23:38:47.118034 5201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c362aad0574cceb808d74fcc19c592" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40449 } }
I20260810 23:38:47.118271 5201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:47.118378 5201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:47.118515 5201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:47.120772 5201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c362aad0574cceb808d74fcc19c592" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40449 } }
I20260810 23:38:47.121291 5201 leader_election.cc:302] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c362aad0574cceb808d74fcc19c592; no voters:
I20260810 23:38:47.121938 5201 leader_election.cc:288] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:47.122093 5213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:47.122825 5213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [term 1 LEADER]: Becoming Leader. State: Replica: 53c362aad0574cceb808d74fcc19c592, State: Running, Role: LEADER
I20260810 23:38:47.123445 5213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c362aad0574cceb808d74fcc19c592" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40449 } }
I20260810 23:38:47.124143 5201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:47.134068 5216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c362aad0574cceb808d74fcc19c592. Latest consensus state: current_term: 1 leader_uuid: "53c362aad0574cceb808d74fcc19c592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c362aad0574cceb808d74fcc19c592" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40449 } } }
I20260810 23:38:47.134581 5216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:47.134419 5214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c362aad0574cceb808d74fcc19c592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c362aad0574cceb808d74fcc19c592" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40449 } } }
I20260810 23:38:47.134881 5214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:47.136512 5226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:47.142526 5226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:47.153283 5226 catalog_manager.cc:1347] Generated new cluster ID: bc81c77f4b9a45a4849088fcc669d1ab
I20260810 23:38:47.153537 5226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:47.172410 5226 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:47.174137 5226 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:47.190853 5226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c362aad0574cceb808d74fcc19c592: Generated new TSK 0
I20260810 23:38:47.191829 5226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:47.259855 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40449
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:47.611128 5242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:47.611438 5242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:47.611598 5242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:47.619167 5242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:47.619304 5242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:47.619450 5242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:47.630039 5242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:47.631450 5242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:47.632900 5242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:47.647763 5252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:47.647809 5251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:47.647763 5255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:47.648777 5242 server_base.cc:1034] running on GCE node
I20260810 23:38:47.649495 5242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:47.650655 5242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:47.651906 5242 hybrid_clock.cc:648] HybridClock initialized: now 1786405127651828 us; error 62 us; skew 500 ppm
I20260810 23:38:47.652276 5242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:47 dist-test-slave-pww0 krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405127, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:47.655824 5242 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:47.657567 5242 webserver.cc:460] Webserver started at http://127.30.33.1:42575/ using document root <none> and password file <none>
I20260810 23:38:47.658361 5242 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:47.658500 5242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:47.658842 5242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:47.662189 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "15805625ce3e41caac66d504d0fb4c78"
format_stamp: "Formatted at 2026-08-10 23:38:47 on dist-test-slave-pww0"
server_key: "32c252c792aac813b849a3b0546c6707"
server_key_iv: "9f32792380f8808db311f1e687e563d4"
server_key_version: "encryptionkey@0"
I20260810 23:38:47.663079 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "15805625ce3e41caac66d504d0fb4c78"
format_stamp: "Formatted at 2026-08-10 23:38:47 on dist-test-slave-pww0"
server_key: "32c252c792aac813b849a3b0546c6707"
server_key_iv: "9f32792380f8808db311f1e687e563d4"
server_key_version: "encryptionkey@0"
I20260810 23:38:47.669077 5242 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 23:38:47.673164 5267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:47.674737 5242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:47.674929 5242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "15805625ce3e41caac66d504d0fb4c78"
format_stamp: "Formatted at 2026-08-10 23:38:47 on dist-test-slave-pww0"
server_key: "32c252c792aac813b849a3b0546c6707"
server_key_iv: "9f32792380f8808db311f1e687e563d4"
server_key_version: "encryptionkey@0"
I20260810 23:38:47.675163 5242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:47.693125 5242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:47.698050 5242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:47.698355 5242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:47.699440 5242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:47.701010 5242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:47.701107 5242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:47.701241 5242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:47.701297 5242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:47.733865 5242 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:45725
I20260810 23:38:47.733886 5466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:45725 every 8 connection(s)
I20260810 23:38:47.735351 5242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:47.736372 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5242
I20260810 23:38:47.736629 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:47.737486 30852 external_mini_cluster.cc:1428] Setting key 18e878edb880e2399263899a7e464d2d
Aug 10 23:38:47 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405127, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:47.759631 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:47.738512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:35719 (local address 127.30.33.62:40449)
0810 23:38:47.738995 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:47.739001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:47.740173 (+ 1172us) server_negotiation.cc:407] Connection header received
0810 23:38:47.742706 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:47.742711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:47.742784 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:47.742880 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:47.745018 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:47.745814 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:47.748081 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:47.748517 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:47.748642 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:47.752650 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:47.752658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:47.752663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:47.752714 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:47.756020 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:47.756967 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:47.756973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:47.756976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:47.757139 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:47.757576 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:47.757581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:47.757584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:47.758009 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:47.758166 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:47.758933 (+ 767us) server_negotiation.cc:299] Negotiation successful
0810 23:38:47.759195 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260810 23:38:47.760483 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:47.739113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40449 (local address 127.30.33.1:35719)
0810 23:38:47.739977 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:47.740039 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:47.742377 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:47.743162 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:47.743174 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:47.743940 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:47.744794 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:47.744820 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:47.746000 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:47.746005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:47.747889 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:47.747906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:47.748788 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:47.748795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:47.749097 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:47.749906 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:47.749935 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:47.752303 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:47.756241 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:47.756250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:47.756266 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:47.756799 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:47.757243 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:47.757247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:47.757249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:47.757453 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:47.758125 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:47.758131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:47.758509 (+ 378us) client_negotiation.cc:769] Sending connection context
0810 23:38:47.758763 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 23:38:47.759206 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260810 23:38:47.762161 5467 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40449
I20260810 23:38:47.762485 5467 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:47.763266 5467 heartbeater.cc:511] Master 127.30.33.62:40449 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
I20260810 23:38:47.765648 5144 ts_manager.cc:194] Registered new tserver with Master: 15805625ce3e41caac66d504d0fb4c78 (127.30.33.1:45725)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
I20260810 23:38:47.767606 5144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:35719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:47.795524 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40449
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:48.158447 5477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:48.159003 5477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:48.159235 5477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:48.170637 5477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:48.170795 5477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:48.170971 5477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:48.181620 5477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:48.183110 5477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:48.184588 5477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:48.198069 5494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:48.198307 5489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:48.198310 5488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:48.199600 5477 server_base.cc:1034] running on GCE node
I20260810 23:38:48.200234 5477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:48.201242 5477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:48.202477 5477 hybrid_clock.cc:648] HybridClock initialized: now 1786405128202407 us; error 53 us; skew 500 ppm
I20260810 23:38:48.202829 5477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:48 dist-test-slave-pww0 krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:48.206203 5477 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:48.207800 5477 webserver.cc:460] Webserver started at http://127.30.33.2:44469/ using document root <none> and password file <none>
I20260810 23:38:48.208510 5477 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:48.208652 5477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:48.208988 5477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:48.212158 5477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "4e59a167723745a39f62fe1eb3dda726"
format_stamp: "Formatted at 2026-08-10 23:38:48 on dist-test-slave-pww0"
server_key: "8c0e9ff4da7393e14597684be9427539"
server_key_iv: "bf9ab3eef6b0b5e988025e822aa2914e"
server_key_version: "encryptionkey@0"
I20260810 23:38:48.212999 5477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "4e59a167723745a39f62fe1eb3dda726"
format_stamp: "Formatted at 2026-08-10 23:38:48 on dist-test-slave-pww0"
server_key: "8c0e9ff4da7393e14597684be9427539"
server_key_iv: "bf9ab3eef6b0b5e988025e822aa2914e"
server_key_version: "encryptionkey@0"
I20260810 23:38:48.218631 5477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:38:48.222625 5508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:48.224242 5477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:48.224453 5477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "4e59a167723745a39f62fe1eb3dda726"
format_stamp: "Formatted at 2026-08-10 23:38:48 on dist-test-slave-pww0"
server_key: "8c0e9ff4da7393e14597684be9427539"
server_key_iv: "bf9ab3eef6b0b5e988025e822aa2914e"
server_key_version: "encryptionkey@0"
I20260810 23:38:48.224673 5477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:48.252051 5477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:48.262323 5477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:48.262646 5477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:48.263778 5477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:48.265354 5477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:48.265436 5477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:48.265558 5477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:48.265619 5477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:48.297720 5477 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:45599
I20260810 23:38:48.297770 5695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:45599 every 8 connection(s)
I20260810 23:38:48.299101 5477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:48.299556 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5477
I20260810 23:38:48.299780 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:48.300341 30852 external_mini_cluster.cc:1428] Setting key a624b5def059b9cb6fbd4261c3685f13
Aug 10 23:38:48 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:48.323313 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:48.302324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:34091 (local address 127.30.33.62:40449)
0810 23:38:48.306913 (+ 4589us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:48.306919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:48.306938 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:38:48.307000 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:48.307005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:48.307081 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:48.307189 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:48.308926 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.309931 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.311592 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.312195 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.312298 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:48.316587 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:48.316592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:48.316595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:48.316634 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:48.319394 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.320333 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.320338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.320340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.320508 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.321091 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.321099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.321103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.321542 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:48.321725 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:48.322589 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 23:38:48.322889 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4457,"thread_start_us":290,"threads_started":1}
I20260810 23:38:48.324150 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:48.303039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40449 (local address 127.30.33.2:34091)
0810 23:38:48.303947 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:48.304014 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:48.306328 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:48.307447 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:48.307487 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:48.308083 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:48.308705 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:48.308725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.310084 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.310088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:48.311406 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:48.311421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.312436 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.312444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:48.312783 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:48.313566 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:48.313599 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:48.316296 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:48.319611 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:48.319622 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:48.319645 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:48.320167 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:48.320641 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:48.320646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:48.320650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:48.320888 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:48.321728 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:48.321735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:48.322123 (+ 388us) client_negotiation.cc:769] Sending connection context
0810 23:38:48.322415 (+ 292us) client_negotiation.cc:240] Negotiation successful
0810 23:38:48.322889 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":347,"threads_started":1}
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
I20260810 23:38:48.326871 5696 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40449
I20260810 23:38:48.327204 5696 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:48.328073 5696 heartbeater.cc:511] Master 127.30.33.62:40449 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
I20260810 23:38:48.329698 5144 ts_manager.cc:194] Registered new tserver with Master: 4e59a167723745a39f62fe1eb3dda726 (127.30.33.2:45599)
I20260810 23:38:48.330585 5144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:34091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:48.359505 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40449
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:48.703181 5706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:48.703531 5706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:48.703687 5706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:48.711093 5706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:48.711237 5706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:48.711400 5706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:48.721453 5706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:48.722882 5706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:48.724398 5706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:48.738776 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:48.738694 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:48.739056 5717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:48.739527 5706 server_base.cc:1034] running on GCE node
I20260810 23:38:48.740234 5706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:48.741207 5706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:48.742426 5706 hybrid_clock.cc:648] HybridClock initialized: now 1786405128742361 us; error 57 us; skew 500 ppm
I20260810 23:38:48.742772 5706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:48 dist-test-slave-pww0 krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:48.745963 5706 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:48.747449 5706 webserver.cc:460] Webserver started at http://127.30.33.3:36713/ using document root <none> and password file <none>
I20260810 23:38:48.748195 5706 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:48.748318 5706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:48.748636 5706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:48.751783 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "a17b8522b3b347f192195eba2820fb41"
format_stamp: "Formatted at 2026-08-10 23:38:48 on dist-test-slave-pww0"
server_key: "53041ed56cf6aa2d4366d20847e5ff35"
server_key_iv: "e4d95ae4ef6d1b8c0e12edbbfafc09e6"
server_key_version: "encryptionkey@0"
I20260810 23:38:48.752591 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "a17b8522b3b347f192195eba2820fb41"
format_stamp: "Formatted at 2026-08-10 23:38:48 on dist-test-slave-pww0"
server_key: "53041ed56cf6aa2d4366d20847e5ff35"
server_key_iv: "e4d95ae4ef6d1b8c0e12edbbfafc09e6"
server_key_version: "encryptionkey@0"
I20260810 23:38:48.758173 5706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:38:48.761950 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:48.763509 5706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:38:48.763713 5706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "a17b8522b3b347f192195eba2820fb41"
format_stamp: "Formatted at 2026-08-10 23:38:48 on dist-test-slave-pww0"
server_key: "53041ed56cf6aa2d4366d20847e5ff35"
server_key_iv: "e4d95ae4ef6d1b8c0e12edbbfafc09e6"
server_key_version: "encryptionkey@0"
I20260810 23:38:48.763936 5706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:48.770790 5467 heartbeater.cc:503] Master 127.30.33.62:40449 was elected leader, sending a full tablet report...
I20260810 23:38:48.786864 5706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:48.791352 5706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:48.791666 5706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:48.792682 5706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:48.794245 5706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:48.794330 5706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:48.794454 5706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:48.794507 5706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:48.825840 5706 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:34185
I20260810 23:38:48.825913 5959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:34185 every 8 connection(s)
I20260810 23:38:48.827170 5706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:48.835291 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5706
I20260810 23:38:48.835534 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:48.836020 30852 external_mini_cluster.cc:1428] Setting key 792e34ff46dc8007694cf8226dcfd51f
Aug 10 23:38:48 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:48.848316 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:48.830017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:52333 (local address 127.30.33.62:40449)
0810 23:38:48.830542 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:48.830550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:48.831700 (+ 1150us) server_negotiation.cc:407] Connection header received
0810 23:38:48.833520 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:48.833525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:48.833609 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:48.833712 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:48.835442 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.836344 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.838297 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.838751 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.838876 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:48.842432 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:48.842438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:48.842441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:48.842477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:48.845243 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.846079 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.846083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.846085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.846219 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.846651 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.846654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.846656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.846931 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:48.847043 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:48.847821 (+ 778us) server_negotiation.cc:299] Negotiation successful
0810 23:38:48.848013 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":222,"threads_started":1}
I20260810 23:38:48.849288 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:48.830726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40449 (local address 127.30.33.3:52333)
0810 23:38:48.831548 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:48.831589 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:48.833238 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:48.833928 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:48.833937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:48.834497 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:48.835226 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:48.835262 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.836538 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.836544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:48.838160 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:48.838173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.838994 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.839002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:48.839288 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:48.839967 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:48.839996 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:48.842202 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:48.845418 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:48.845427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:48.845442 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:48.845954 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:48.846358 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:48.846362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:48.846364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:48.846568 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:48.847026 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:48.847032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:48.847389 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 23:38:48.847713 (+ 324us) client_negotiation.cc:240] Negotiation successful
0810 23:38:48.848152 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":291,"threads_started":1}
I20260810 23:38:48.851042 5960 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40449
I20260810 23:38:48.851366 5960 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:48.852245 5960 heartbeater.cc:511] Master 127.30.33.62:40449 requested a full tablet report, sending...
I20260810 23:38:48.853808 5142 ts_manager.cc:194] Registered new tserver with Master: a17b8522b3b347f192195eba2820fb41 (127.30.33.3:34185)
I20260810 23:38:48.854748 5142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:52333
I20260810 23:38:48.864800 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:38:48 dist-test-slave-pww0 krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:38:48 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:48.923734 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:48.907272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.30.33.1:45725)
0810 23:38:48.907930 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:48.907937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:48.907967 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:38:48.908154 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:48.908163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:48.908395 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:48.908535 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:48.910149 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.911254 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.912494 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.913102 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.913201 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:48.917361 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:48.917381 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:48.917397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:48.917448 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:48.920414 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.921233 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.921243 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.921245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.921401 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.921866 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.921871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.921873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.922291 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:48.922611 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:48.922990 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:38:48.923302 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":328,"threads_started":1}
Aug 10 23:38:48 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:48.942682 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:48.928793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.30.33.62:40449)
0810 23:38:48.929014 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:48.929020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:48.929253 (+ 233us) server_negotiation.cc:407] Connection header received
0810 23:38:48.929427 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:48.929431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:48.929497 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:48.929569 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:48.930889 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.931723 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.932920 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.933331 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.933428 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:48.937419 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:48.937424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:48.937427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:48.937465 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:48.940027 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.940710 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.940715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.940717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.940846 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.941213 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.941217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.941219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.941557 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:48.941817 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:48.942124 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:38:48.942359 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 23:38:48.943333 5142 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34590
I20260810 23:38:48.944451 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5242
I20260810 23:38:48.957701 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:48.945389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.30.33.1:45725)
0810 23:38:48.945633 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:48.945640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:48.945836 (+ 196us) server_negotiation.cc:407] Connection header received
0810 23:38:48.946009 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:48.946014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:48.946081 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:48.946167 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:48.947375 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.948128 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.949357 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:48.949815 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:48.949930 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:48.952350 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:48.952356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:48.952359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:48.952398 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:48.954997 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.955768 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.955774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.955777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.955911 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:48.956288 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:48.956293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:48.956295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:48.956575 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:48.956816 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:48.957133 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 23:38:48.957369 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:38:48.958379 5461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:49.044198 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5242
I20260810 23:38:49.065115 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5477
Aug 10 23:38:49 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:38:49.081904 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:49.066104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.30.33.2:45599)
0810 23:38:49.066666 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:49.066674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:49.066702 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:38:49.066783 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:49.066789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:49.067010 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:49.067138 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:49.068425 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:49.069525 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:49.070863 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:49.071443 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:49.071582 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:49.075766 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:49.075782 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:49.075797 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:49.075837 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:49.078933 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:49.079840 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:49.079848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:49.079851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:49.079983 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:49.080341 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:49.080344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:49.080346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:49.080716 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:49.080959 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:49.081286 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 23:38:49.081518 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":284,"threads_started":1}
I20260810 23:38:49.083292 5688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:49.170501 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5477
I20260810 23:38:49.197324 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5706
Aug 10 23:38:49 dist-test-slave-pww0 krb5kdc[5059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:38:49.215147 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:49.198460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.30.33.3:34185)
0810 23:38:49.199164 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:49.199171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:49.199202 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:38:49.199268 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:49.199276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:49.199538 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:49.199686 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:49.201021 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:49.202121 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:49.203413 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:49.204155 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:49.204273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:49.209078 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:49.209097 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:49.209115 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:49.209163 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:49.212136 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:49.212944 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:49.212952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:49.212954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:49.213102 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:49.213545 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:49.213549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:49.213551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:49.213973 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:49.214226 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:49.214562 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 23:38:49.214818 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":377,"threads_started":1}
I20260810 23:38:49.216555 5950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:49.299578 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5706
I20260810 23:38:49.322252 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5085
I20260810 23:38:49.323572 5191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:49.430473 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5085
2026-08-10T23:38:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4340 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:49 dist-test-slave-pww0 krb5kdc[5991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:49 dist-test-slave-pww0 krb5kdc[5991](info): set up 2 sockets
Aug 10 23:38:49 dist-test-slave-pww0 krb5kdc[5991](info): commencing operation
krb5kdc: starting...
W20260810 23:38:50.706525 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:38:50 dist-test-slave-pww0 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:38:50.967322 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37995
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:51.318542 6013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:51.318897 6013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:51.319046 6013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:51.326669 6013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:51.326818 6013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:51.326900 6013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:51.326977 6013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:51.327059 6013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:51.337527 6013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:51.338919 6013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:51.340313 6013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:51.352370 6021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:51.352437 6023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:51.352459 6025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:51.353163 6013 server_base.cc:1034] running on GCE node
I20260810 23:38:51.354060 6013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:51.355618 6013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:51.356829 6013 hybrid_clock.cc:648] HybridClock initialized: now 1786405131356781 us; error 47 us; skew 500 ppm
I20260810 23:38:51.357191 6013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:51 dist-test-slave-pww0 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405131, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:51.364693 6013 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:38:51.366006 6013 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:51.366134 6013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:51.366501 6013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:51.369854 6013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "630df8d366564df08513db52fe03c7bc"
format_stamp: "Formatted at 2026-08-10 23:38:51 on dist-test-slave-pww0"
server_key: "58c936410e7aaaa39e34b1ec37ed8fb8"
server_key_iv: "1315bcec6ac34da53f6e7130cad34b41"
server_key_version: "encryptionkey@0"
I20260810 23:38:51.370738 6013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "630df8d366564df08513db52fe03c7bc"
format_stamp: "Formatted at 2026-08-10 23:38:51 on dist-test-slave-pww0"
server_key: "58c936410e7aaaa39e34b1ec37ed8fb8"
server_key_iv: "1315bcec6ac34da53f6e7130cad34b41"
server_key_version: "encryptionkey@0"
I20260810 23:38:51.376288 6013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:38:51.380216 6034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:51.381779 6013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:51.381980 6013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "630df8d366564df08513db52fe03c7bc"
format_stamp: "Formatted at 2026-08-10 23:38:51 on dist-test-slave-pww0"
server_key: "58c936410e7aaaa39e34b1ec37ed8fb8"
server_key_iv: "1315bcec6ac34da53f6e7130cad34b41"
server_key_version: "encryptionkey@0"
I20260810 23:38:51.382222 6013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:51.400090 6013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:51.404719 6013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:51.405001 6013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:51.423552 6013 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:37995
I20260810 23:38:51.423547 6131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:37995 every 8 connection(s)
I20260810 23:38:51.424608 6013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:51.428639 6132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:51.434036 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6013
I20260810 23:38:51.434278 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:51.434979 30852 external_mini_cluster.cc:1428] Setting key 72e31c6b24508089b41e9bc61dc7a592
I20260810 23:38:51.441043 6132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc: Bootstrap starting.
I20260810 23:38:51.444960 6132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:51.446182 6132 log.cc:824] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:51.449095 6132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc: No bootstrap required, opened a new log
Aug 10 23:38:51 dist-test-slave-pww0 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:51.457381 6132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630df8d366564df08513db52fe03c7bc" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37995 } }
I20260810 23:38:51.457794 6132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:51.457908 6132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630df8d366564df08513db52fe03c7bc, State: Initialized, Role: FOLLOWER
I20260810 23:38:51.458602 6132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630df8d366564df08513db52fe03c7bc" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37995 } }
I20260810 23:38:51.458854 6132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:51.459039 6132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:51.459301 6132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:51.460505 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:51.437270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.30.33.62:37995)
0810 23:38:51.438204 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:51.438222 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:51.438284 (+ 62us) server_negotiation.cc:407] Connection header received
0810 23:38:51.440770 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:51.440804 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:51.441382 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:51.441874 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:51.443348 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:51.445376 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:51.446880 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:51.447664 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:51.447860 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:51.452220 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:51.452230 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:51.452252 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:51.452315 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:51.455776 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:51.456609 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:51.456618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:51.456626 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:51.456835 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:51.457265 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:51.457269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:51.457271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:51.457995 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:51.458309 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:51.458785 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 23:38:51.459297 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":394,"threads_started":1}
I20260810 23:38:51.461900 6132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630df8d366564df08513db52fe03c7bc" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37995 } }
I20260810 23:38:51.462460 6132 leader_election.cc:302] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 630df8d366564df08513db52fe03c7bc; no voters:
I20260810 23:38:51.463054 6132 leader_election.cc:288] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:51.463239 6145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:51.464358 6145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [term 1 LEADER]: Becoming Leader. State: Replica: 630df8d366564df08513db52fe03c7bc, State: Running, Role: LEADER
I20260810 23:38:51.464881 6145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630df8d366564df08513db52fe03c7bc" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37995 } }
I20260810 23:38:51.465212 6132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:51.467716 6150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "630df8d366564df08513db52fe03c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630df8d366564df08513db52fe03c7bc" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37995 } } }
I20260810 23:38:51.468116 6150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:51.468673 6151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 630df8d366564df08513db52fe03c7bc. Latest consensus state: current_term: 1 leader_uuid: "630df8d366564df08513db52fe03c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "630df8d366564df08513db52fe03c7bc" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37995 } } }
I20260810 23:38:51.469002 6151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:51.474915 6157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:51.480815 6157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:51.490343 6157 catalog_manager.cc:1347] Generated new cluster ID: 80b9c3598f0046719c28ed072da73a2f
I20260810 23:38:51.490480 6157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:51.525777 6157 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:51.527223 6157 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:51.540648 6157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc: Generated new TSK 0
I20260810 23:38:51.541606 6157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:38:51.615705 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37995
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:51.968400 6185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:51.968712 6185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:51.968822 6185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:51.976742 6185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:51.976899 6185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:51.977064 6185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:51.987839 6185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:51.989285 6185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:51.990765 6185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:52.004832 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:52.004815 6200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:52.004877 6198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:52.005535 6185 server_base.cc:1034] running on GCE node
I20260810 23:38:52.006250 6185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:52.007261 6185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:52.008471 6185 hybrid_clock.cc:648] HybridClock initialized: now 1786405132008417 us; error 59 us; skew 500 ppm
I20260810 23:38:52.008805 6185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:52 dist-test-slave-pww0 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405132, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:52.012373 6185 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:38:52.013608 6185 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:52.013725 6185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:52.014079 6185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:52.017386 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "9b364bc98b1d447a9ab5fc069a668d2a"
format_stamp: "Formatted at 2026-08-10 23:38:52 on dist-test-slave-pww0"
server_key: "902e260801e19dd2b1f98e9ba22e1773"
server_key_iv: "b249c472e587a8744d71ffa51c457162"
server_key_version: "encryptionkey@0"
I20260810 23:38:52.018268 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "9b364bc98b1d447a9ab5fc069a668d2a"
format_stamp: "Formatted at 2026-08-10 23:38:52 on dist-test-slave-pww0"
server_key: "902e260801e19dd2b1f98e9ba22e1773"
server_key_iv: "b249c472e587a8744d71ffa51c457162"
server_key_version: "encryptionkey@0"
I20260810 23:38:52.024009 6185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:38:52.028050 6210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:52.029618 6185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:38:52.029820 6185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "9b364bc98b1d447a9ab5fc069a668d2a"
format_stamp: "Formatted at 2026-08-10 23:38:52 on dist-test-slave-pww0"
server_key: "902e260801e19dd2b1f98e9ba22e1773"
server_key_iv: "b249c472e587a8744d71ffa51c457162"
server_key_version: "encryptionkey@0"
I20260810 23:38:52.030112 6185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:52.049925 6185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:52.054519 6185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:52.054819 6185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:52.056015 6185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:52.057538 6185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:52.057652 6185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:52.057768 6185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:52.057840 6185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:52.091830 6185 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:38001
I20260810 23:38:52.091881 6410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:38001 every 8 connection(s)
I20260810 23:38:52.092764 6185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:52.101754 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6185
I20260810 23:38:52.101994 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:52.102681 30852 external_mini_cluster.cc:1428] Setting key ba040c222bcbb7f89bd3a4b188043d59
Aug 10 23:38:52 dist-test-slave-pww0 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405132, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:52.116025 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:52.095301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:45965 (local address 127.30.33.62:37995)
0810 23:38:52.095810 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:52.095816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:52.096860 (+ 1044us) server_negotiation.cc:407] Connection header received
0810 23:38:52.098807 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:52.098812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:52.098906 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:52.099018 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:52.100968 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.102022 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.103966 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.104441 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.104599 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:52.108872 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:52.108879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:52.108883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:52.108931 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:52.112172 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:52.113119 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:52.113125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:52.113128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:52.113306 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:52.113880 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:52.113885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:52.113888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:52.114259 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:52.114408 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:52.115333 (+ 925us) server_negotiation.cc:299] Negotiation successful
0810 23:38:52.115648 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":246,"threads_started":1}
I20260810 23:38:52.117084 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:52.095920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37995 (local address 127.30.33.1:45965)
0810 23:38:52.096684 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:52.096727 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:52.098486 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:52.099250 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:52.099260 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:52.100084 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:52.100750 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:52.100776 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.102212 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.102216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:52.103812 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:52.103829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.104760 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.104770 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:52.105062 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:52.105884 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:52.105910 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:52.108528 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:52.112393 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:52.112402 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:52.112418 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:52.112960 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:52.113465 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:52.113469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:52.113472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:52.113740 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:52.114406 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:52.114420 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:52.114853 (+ 433us) client_negotiation.cc:769] Sending connection context
0810 23:38:52.115139 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 23:38:52.115817 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":326,"threads_started":1}
I20260810 23:38:52.118985 6414 heartbeater.cc:347] Connected to a master server at 127.30.33.62:37995
I20260810 23:38:52.119319 6414 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:52.120092 6414 heartbeater.cc:511] Master 127.30.33.62:37995 requested a full tablet report, sending...
I20260810 23:38:52.122679 6067 ts_manager.cc:194] Registered new tserver with Master: 9b364bc98b1d447a9ab5fc069a668d2a (127.30.33.1:38001)
I20260810 23:38:52.124774 6067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:45965
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:38:52.162068 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37995
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:52.510196 6421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:52.510540 6421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:52.510723 6421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:52.518299 6421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:52.518433 6421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:52.518555 6421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:52.529017 6421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:52.530440 6421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:52.531863 6421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:52.546185 6428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:52.546123 6432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:52.546075 6429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:52.547122 6421 server_base.cc:1034] running on GCE node
I20260810 23:38:52.547861 6421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:52.549036 6421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:52.550271 6421 hybrid_clock.cc:648] HybridClock initialized: now 1786405132550194 us; error 67 us; skew 500 ppm
I20260810 23:38:52.550635 6421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:52 dist-test-slave-pww0 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405132, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:52.554208 6421 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:38:52.555608 6421 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:52.555734 6421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:52.556080 6421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:52.559422 6421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "5b186ba15bf345a2a8c0d28d0bec457a"
format_stamp: "Formatted at 2026-08-10 23:38:52 on dist-test-slave-pww0"
server_key: "42752c764710a1ac55b82513e9694d0f"
server_key_iv: "1d625152f475b6a08a73f2200909c58f"
server_key_version: "encryptionkey@0"
I20260810 23:38:52.560427 6421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "5b186ba15bf345a2a8c0d28d0bec457a"
format_stamp: "Formatted at 2026-08-10 23:38:52 on dist-test-slave-pww0"
server_key: "42752c764710a1ac55b82513e9694d0f"
server_key_iv: "1d625152f475b6a08a73f2200909c58f"
server_key_version: "encryptionkey@0"
I20260810 23:38:52.566519 6421 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:38:52.570731 6442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:52.572428 6421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:52.572623 6421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "5b186ba15bf345a2a8c0d28d0bec457a"
format_stamp: "Formatted at 2026-08-10 23:38:52 on dist-test-slave-pww0"
server_key: "42752c764710a1ac55b82513e9694d0f"
server_key_iv: "1d625152f475b6a08a73f2200909c58f"
server_key_version: "encryptionkey@0"
I20260810 23:38:52.572891 6421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:52.585757 6421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:52.591004 6421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:52.591332 6421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:52.592519 6421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:52.594422 6421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:52.594535 6421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:52.594671 6421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:52.594745 6421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:52.628567 6421 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:38003
I20260810 23:38:52.628635 6629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:38003 every 8 connection(s)
I20260810 23:38:52.629482 6421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:52.638195 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6421
I20260810 23:38:52.638382 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:52.638825 30852 external_mini_cluster.cc:1428] Setting key 685f065c6d3a8b867f920f39c3436725
Aug 10 23:38:52 dist-test-slave-pww0 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405132, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:52.651669 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:52.632150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:34199 (local address 127.30.33.62:37995)
0810 23:38:52.632614 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:52.632620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:52.633683 (+ 1063us) server_negotiation.cc:407] Connection header received
0810 23:38:52.635822 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:52.635828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:52.635920 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:52.636025 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:52.637769 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.638561 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.640561 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.641034 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.641156 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:52.645241 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:52.645247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:52.645251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:52.645292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:52.648055 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:52.648970 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:52.648975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:52.648978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:52.649128 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:52.649643 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:52.649648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:52.649650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:52.649970 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:52.650123 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:52.650987 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 23:38:52.651254 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":217,"threads_started":1}
I20260810 23:38:52.652810 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:52.632688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37995 (local address 127.30.33.2:34199)
0810 23:38:52.633495 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:52.633537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:52.635481 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:52.636308 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:52.636317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:52.636913 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:52.637556 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:52.637576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.638741 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.638746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:52.640383 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:52.640401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:52.641312 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:52.641322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:52.641551 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:52.642249 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:52.642276 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:52.644928 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:52.648259 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:52.648270 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:52.648286 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:52.648810 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:52.649263 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:52.649270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:52.649273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:52.649507 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:52.650118 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:52.650128 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:52.650561 (+ 433us) client_negotiation.cc:769] Sending connection context
0810 23:38:52.650841 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 23:38:52.651502 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":313,"threads_started":1}
I20260810 23:38:52.654635 6631 heartbeater.cc:347] Connected to a master server at 127.30.33.62:37995
I20260810 23:38:52.654973 6631 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:52.655784 6631 heartbeater.cc:511] Master 127.30.33.62:37995 requested a full tablet report, sending...
I20260810 23:38:52.657466 6067 ts_manager.cc:194] Registered new tserver with Master: 5b186ba15bf345a2a8c0d28d0bec457a (127.30.33.2:38003)
I20260810 23:38:52.658777 6067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:34199
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:38:52.696766 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37995
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:53.044574 6638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:53.044984 6638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:53.045142 6638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:53.052938 6638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:53.053064 6638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:53.053223 6638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:53.063508 6638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:53.064970 6638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:53.066404 6638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:53.079990 6645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:53.079962 6647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:53.080015 6649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:53.080780 6638 server_base.cc:1034] running on GCE node
I20260810 23:38:53.081441 6638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:53.082501 6638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:53.083705 6638 hybrid_clock.cc:648] HybridClock initialized: now 1786405133083646 us; error 55 us; skew 500 ppm
I20260810 23:38:53.084041 6638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:53 dist-test-slave-pww0 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405133, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:53.087695 6638 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:38:53.088966 6638 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:53.089077 6638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:53.089408 6638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:53.092684 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "ff75ad128e03452a81bb1d8b42e8ba4e"
format_stamp: "Formatted at 2026-08-10 23:38:53 on dist-test-slave-pww0"
server_key: "05b4747e87f386876ef91dc2943f724e"
server_key_iv: "1d85843a415ca1d11868c4a68ce70121"
server_key_version: "encryptionkey@0"
I20260810 23:38:53.093524 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "ff75ad128e03452a81bb1d8b42e8ba4e"
format_stamp: "Formatted at 2026-08-10 23:38:53 on dist-test-slave-pww0"
server_key: "05b4747e87f386876ef91dc2943f724e"
server_key_iv: "1d85843a415ca1d11868c4a68ce70121"
server_key_version: "encryptionkey@0"
I20260810 23:38:53.099088 6638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:38:53.103029 6663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:53.104549 6638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:53.104763 6638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "ff75ad128e03452a81bb1d8b42e8ba4e"
format_stamp: "Formatted at 2026-08-10 23:38:53 on dist-test-slave-pww0"
server_key: "05b4747e87f386876ef91dc2943f724e"
server_key_iv: "1d85843a415ca1d11868c4a68ce70121"
server_key_version: "encryptionkey@0"
I20260810 23:38:53.105000 6638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:53.129212 6414 heartbeater.cc:503] Master 127.30.33.62:37995 was elected leader, sending a full tablet report...
I20260810 23:38:53.130857 6638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:53.136111 6638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:53.136392 6638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:53.137444 6638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:53.138900 6638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:53.138995 6638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:53.139107 6638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:53.139173 6638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:53.170125 6638 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:42879
I20260810 23:38:53.170185 6903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:42879 every 8 connection(s)
I20260810 23:38:53.171027 6638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:53.173442 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6638
I20260810 23:38:53.173658 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:53.174139 30852 external_mini_cluster.cc:1428] Setting key 2f9e5e54add9acad44d337e8be155864
Aug 10 23:38:53 dist-test-slave-pww0 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405133, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:38:53.192873 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.173855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:43329 (local address 127.30.33.62:37995)
0810 23:38:53.174410 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.174416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.175514 (+ 1098us) server_negotiation.cc:407] Connection header received
0810 23:38:53.177909 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.177914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.178013 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.178110 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:53.179790 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.180592 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.182177 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.182592 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.182717 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.186299 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:53.186304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:53.186307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:53.186349 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:53.189656 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.190479 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.190485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.190488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.190615 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.191045 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.191050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.191052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.191330 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:53.191507 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.192256 (+ 749us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.192482 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":266,"threads_started":1}
I20260810 23:38:53.193899 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.174471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37995 (local address 127.30.33.3:43329)
0810 23:38:53.175342 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:53.175391 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:53.177632 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:53.178377 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:53.178386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:53.178935 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:53.179618 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.179641 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.180750 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.180754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.182020 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.182033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.182829 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.182837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.183064 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.183773 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:53.183797 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:53.186051 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:53.189810 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:53.189820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:53.189840 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:53.190353 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:53.190719 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:53.190724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:53.190728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:53.190926 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:53.191492 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:53.191501 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:53.191869 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 23:38:53.192125 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 23:38:53.192723 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":338,"threads_started":1}
I20260810 23:38:53.195443 6904 heartbeater.cc:347] Connected to a master server at 127.30.33.62:37995
I20260810 23:38:53.195829 6904 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:53.196547 6904 heartbeater.cc:511] Master 127.30.33.62:37995 requested a full tablet report, sending...
I20260810 23:38:53.198136 6067 ts_manager.cc:194] Registered new tserver with Master: ff75ad128e03452a81bb1d8b42e8ba4e (127.30.33.3:42879)
I20260810 23:38:53.198966 6067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:43329
I20260810 23:38:53.202732 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:38:53.219892 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.207371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.30.33.62:37995)
0810 23:38:53.207579 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.207584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.208125 (+ 541us) server_negotiation.cc:407] Connection header received
0810 23:38:53.208277 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.208281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.208348 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.208435 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:53.209971 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.210752 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.212080 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.212498 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.212597 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.214845 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:53.214849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:53.214853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:53.214882 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:53.217380 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.218086 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.218091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.218097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.218247 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.218588 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.218591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.218593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.218937 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:53.219136 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.219392 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.219650 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:38:53.224159 6067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:53.231163 6067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:53.273052 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.243904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.30.33.2:38003)
0810 23:38:53.260179 (+ 16275us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.260190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.260223 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:38:53.260343 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.260352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.260686 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.262113 (+ 1427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:53.263444 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.267792 (+ 4348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.269791 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.270970 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.271077 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.271269 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.272623 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.272801 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16068,"thread_start_us":15977,"threads_started":1}
I20260810 23:38:53.275875 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.244446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.30.33.1:38001)
0810 23:38:53.261436 (+ 16990us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.261445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.261478 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:38:53.261587 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.261595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.261869 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.262108 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:53.263821 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.265512 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.268525 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.270890 (+ 2365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.271060 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.271345 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.275296 (+ 3951us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.275533 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16772,"thread_start_us":16585,"threads_started":1}
I20260810 23:38:53.276345 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.243727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:38003 (local address 127.0.0.1:43420)
0810 23:38:53.253867 (+ 10140us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:53.253917 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:53.254174 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:53.262475 (+ 8301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:53.262484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:53.262519 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:53.263204 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.263222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.267979 (+ 4757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.267984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.269603 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.269619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.271393 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.271399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.271615 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.271659 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:38:53.271774 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:38:53.271959 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9849,"thread_start_us":17168,"threads_started":1}
I20260810 23:38:53.276546 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.243893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:38001 (local address 127.0.0.1:38504)
0810 23:38:53.259410 (+ 15517us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:53.259435 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:53.259615 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:53.262510 (+ 2895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:53.262515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:53.262545 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:53.263572 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.263586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.265775 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.265780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.268325 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.268341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.272090 (+ 3749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.272099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.272394 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.272431 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:38:53.272546 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:38:53.275152 (+ 2606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15339,"mutex_wait_us":2431,"thread_start_us":9515,"threads_started":1}
I20260810 23:38:53.278889 6933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.244257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:42879 (local address 127.0.0.1:51180)
0810 23:38:53.261713 (+ 17456us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:53.261735 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:53.261871 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:53.263570 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:53.263574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:53.263604 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:53.264000 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.264014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.271997 (+ 7983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.272002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.275764 (+ 3762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.275783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.278126 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.278134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.278321 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.278358 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:38:53.278430 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:38:53.278568 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17328,"thread_start_us":17171,"threads_started":1}
I20260810 23:38:53.280040 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.244893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.30.33.3:42879)
0810 23:38:53.260817 (+ 15924us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.260828 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.262881 (+ 2053us) server_negotiation.cc:407] Connection header received
0810 23:38:53.263004 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.263011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.263283 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.264213 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:53.264539 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.267863 (+ 3324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.276004 (+ 8141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.277886 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.279146 (+ 1260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.279410 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.279516 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.279681 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15708,"thread_start_us":15530,"threads_started":1}
I20260810 23:38:53.282433 6513 tablet_service.cc:1467] Processing CreateTablet for tablet 42aec42f098546efbdf8a11b0b1dd375 (DEFAULT_TABLE table=test-table [id=e881edadc68a4b8ba58cb77546631b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:53.282230 6297 tablet_service.cc:1467] Processing CreateTablet for tablet 42aec42f098546efbdf8a11b0b1dd375 (DEFAULT_TABLE table=test-table [id=e881edadc68a4b8ba58cb77546631b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:53.283742 6513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42aec42f098546efbdf8a11b0b1dd375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:53.283871 6297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42aec42f098546efbdf8a11b0b1dd375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:53.283900 6763 tablet_service.cc:1467] Processing CreateTablet for tablet 42aec42f098546efbdf8a11b0b1dd375 (DEFAULT_TABLE table=test-table [id=e881edadc68a4b8ba58cb77546631b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:53.285375 6763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42aec42f098546efbdf8a11b0b1dd375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:53.296283 6938 tablet_bootstrap.cc:492] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Bootstrap starting.
I20260810 23:38:53.300246 6939 tablet_bootstrap.cc:492] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Bootstrap starting.
I20260810 23:38:53.300987 6938 tablet_bootstrap.cc:654] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:53.302526 6938 log.cc:824] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:53.304082 6939 tablet_bootstrap.cc:654] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:53.304929 6947 tablet_bootstrap.cc:492] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Bootstrap starting.
I20260810 23:38:53.305447 6939 log.cc:824] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:53.306198 6938 tablet_bootstrap.cc:492] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: No bootstrap required, opened a new log
I20260810 23:38:53.306538 6938 ts_tablet_manager.cc:1397] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:38:53.308282 6947 tablet_bootstrap.cc:654] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:53.309407 6947 log.cc:824] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:53.312721 6947 tablet_bootstrap.cc:492] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: No bootstrap required, opened a new log
I20260810 23:38:53.313036 6947 ts_tablet_manager.cc:1397] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:38:53.315420 6939 tablet_bootstrap.cc:492] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: No bootstrap required, opened a new log
I20260810 23:38:53.315846 6939 ts_tablet_manager.cc:1397] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260810 23:38:53.315980 6938 raft_consensus.cc:348] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.316418 6938 raft_consensus.cc:374] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:53.316526 6938 raft_consensus.cc:729] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b186ba15bf345a2a8c0d28d0bec457a, State: Initialized, Role: FOLLOWER
I20260810 23:38:53.317134 6938 consensus_queue.cc:260] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.318284 6631 heartbeater.cc:503] Master 127.30.33.62:37995 was elected leader, sending a full tablet report...
I20260810 23:38:53.318994 6938 ts_tablet_manager.cc:1428] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260810 23:38:53.323750 6947 raft_consensus.cc:348] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.324242 6947 raft_consensus.cc:374] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:53.324350 6947 raft_consensus.cc:729] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff75ad128e03452a81bb1d8b42e8ba4e, State: Initialized, Role: FOLLOWER
I20260810 23:38:53.325044 6947 consensus_queue.cc:260] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.326363 6904 heartbeater.cc:503] Master 127.30.33.62:37995 was elected leader, sending a full tablet report...
I20260810 23:38:53.326443 6939 raft_consensus.cc:348] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.326946 6939 raft_consensus.cc:374] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:53.327062 6939 raft_consensus.cc:729] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b364bc98b1d447a9ab5fc069a668d2a, State: Initialized, Role: FOLLOWER
I20260810 23:38:53.327253 6947 ts_tablet_manager.cc:1428] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 23:38:53.327818 6939 consensus_queue.cc:260] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.329708 6939 ts_tablet_manager.cc:1428] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260810 23:38:53.345114 6415 tablet.cc:2369] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:38:53.381425 6632 tablet.cc:2369] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:38:53.422839 6905 tablet.cc:2369] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:38:53.627208 6955 raft_consensus.cc:482] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:53.627538 6955 raft_consensus.cc:504] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.629403 6955 leader_election.cc:288] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b186ba15bf345a2a8c0d28d0bec457a (127.30.33.2:38003), 9b364bc98b1d447a9ab5fc069a668d2a (127.30.33.1:38001)
I20260810 23:38:53.636611 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.629845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:38001 (local address 127.30.33.3:36375)
0810 23:38:53.630284 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:53.630307 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:53.630448 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:53.631098 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:53.631102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:53.631132 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:53.631307 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.631317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.632787 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.632793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.634382 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.634397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.635762 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.635770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.636054 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.636083 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:38:53.636172 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:38:53.636338 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260810 23:38:53.636605 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.630529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:36375 (local address 127.30.33.1:38001)
0810 23:38:53.630746 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.630753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.630779 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:38:53.630868 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.630872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.630949 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.631047 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:53.631491 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.632623 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.634579 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.635536 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.635685 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.635805 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.636223 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.636359 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:38:53.637840 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aec42f098546efbdf8a11b0b1dd375" candidate_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" is_pre_election: true
I20260810 23:38:53.638451 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.630064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:38003 (local address 127.30.33.3:40889)
0810 23:38:53.631081 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:53.631098 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:53.631275 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:53.631658 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:53.631663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:53.631687 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:38:53.632009 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.632018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.633375 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.633381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.635541 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:53.635567 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.637617 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.637624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:53.637803 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.637836 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:38:53.637921 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:38:53.638057 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":764,"threads_started":1}
I20260810 23:38:53.638494 6333 raft_consensus.cc:2454] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff75ad128e03452a81bb1d8b42e8ba4e in term 0.
I20260810 23:38:53.638494 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.630032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:40889 (local address 127.30.33.2:38003)
0810 23:38:53.630714 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.630721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.631168 (+ 447us) server_negotiation.cc:407] Connection header received
0810 23:38:53.631437 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.631443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.631545 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.631627 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:38:53.632160 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.633186 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.635745 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.637059 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.637194 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.637299 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.637988 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.638143 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541}
I20260810 23:38:53.639410 6664 leader_election.cc:302] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b364bc98b1d447a9ab5fc069a668d2a, ff75ad128e03452a81bb1d8b42e8ba4e; no voters:
I20260810 23:38:53.639880 6955 raft_consensus.cc:2781] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:53.640017 6955 raft_consensus.cc:482] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:53.640120 6955 raft_consensus.cc:3037] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:53.639914 6553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aec42f098546efbdf8a11b0b1dd375" candidate_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" is_pre_election: true
I20260810 23:38:53.640470 6553 raft_consensus.cc:2454] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff75ad128e03452a81bb1d8b42e8ba4e in term 0.
I20260810 23:38:53.643002 6955 raft_consensus.cc:504] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.643769 6955 leader_election.cc:288] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [CANDIDATE]: Term 1 election: Requested vote from peers 5b186ba15bf345a2a8c0d28d0bec457a (127.30.33.2:38003), 9b364bc98b1d447a9ab5fc069a668d2a (127.30.33.1:38001)
I20260810 23:38:53.644285 6553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aec42f098546efbdf8a11b0b1dd375" candidate_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b186ba15bf345a2a8c0d28d0bec457a"
I20260810 23:38:53.644381 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aec42f098546efbdf8a11b0b1dd375" candidate_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b364bc98b1d447a9ab5fc069a668d2a"
I20260810 23:38:53.644532 6553 raft_consensus.cc:3037] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:53.644613 6333 raft_consensus.cc:3037] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:53.647951 6553 raft_consensus.cc:2454] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff75ad128e03452a81bb1d8b42e8ba4e in term 1.
I20260810 23:38:53.647951 6333 raft_consensus.cc:2454] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff75ad128e03452a81bb1d8b42e8ba4e in term 1.
I20260810 23:38:53.648541 6664 leader_election.cc:302] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b364bc98b1d447a9ab5fc069a668d2a, ff75ad128e03452a81bb1d8b42e8ba4e; no voters:
I20260810 23:38:53.648888 6955 raft_consensus.cc:2781] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:53.649421 6955 raft_consensus.cc:686] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 1 LEADER]: Becoming Leader. State: Replica: ff75ad128e03452a81bb1d8b42e8ba4e, State: Running, Role: LEADER
I20260810 23:38:53.649979 6955 consensus_queue.cc:237] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } }
I20260810 23:38:53.655112 6066 catalog_manager.cc:5495] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e reported cstate change: term changed from 0 to 1, leader changed from <none> to ff75ad128e03452a81bb1d8b42e8ba4e (127.30.33.3). New cstate: current_term: 1 leader_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff75ad128e03452a81bb1d8b42e8ba4e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42879 } health_report { overall_health: HEALTHY } } }
I20260810 23:38:53.745543 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.739286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.30.33.3:42879)
0810 23:38:53.739545 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.739551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.740056 (+ 505us) server_negotiation.cc:407] Connection header received
0810 23:38:53.740260 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.740264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.740362 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.740499 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:53.740954 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.741880 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.743286 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.743777 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.743895 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.744440 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:53.744989 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:53.745123 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.745243 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.745326 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:38:53.754487 6553 raft_consensus.cc:1261] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 1 FOLLOWER]: Refusing update from remote peer ff75ad128e03452a81bb1d8b42e8ba4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:53.754487 6333 raft_consensus.cc:1261] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 1 FOLLOWER]: Refusing update from remote peer ff75ad128e03452a81bb1d8b42e8ba4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:53.755591 6960 consensus_queue.cc:1035] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b364bc98b1d447a9ab5fc069a668d2a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:53.755993 6955 consensus_queue.cc:1035] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b186ba15bf345a2a8c0d28d0bec457a" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 38003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:53.771600 6968 mvcc.cc:205] Tried to move back new op lower bound from 7317115427847716864 to 7317115427434758144. Current Snapshot: MvccSnapshot[applied={T|T < 7317115427847716864}]
I20260810 23:38:53.784171 6066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:53.784420 6066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:53.788142 6066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 630df8d366564df08513db52fe03c7bc: Sending DeleteTablet for 3 replicas of tablet 42aec42f098546efbdf8a11b0b1dd375
I20260810 23:38:53.789431 6513 tablet_service.cc:1514] Processing DeleteTablet for tablet 42aec42f098546efbdf8a11b0b1dd375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:53 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:43420
I20260810 23:38:53.789433 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 42aec42f098546efbdf8a11b0b1dd375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:53 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:51180
I20260810 23:38:53.790269 6297 tablet_service.cc:1514] Processing DeleteTablet for tablet 42aec42f098546efbdf8a11b0b1dd375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:53 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:38504
I20260810 23:38:53.791077 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6185
I20260810 23:38:53.791131 6993 tablet_replica.cc:330] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: stopping tablet replica
I20260810 23:38:53.791853 6993 raft_consensus.cc:2229] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:53.792603 6993 raft_consensus.cc:2258] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:53.795990 6993 ts_tablet_manager.cc:1905] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:53.803263 6992 tablet_replica.cc:330] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: stopping tablet replica
I20260810 23:38:53.803825 6992 raft_consensus.cc:2229] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:53.804000 6986 tablet_replica.cc:330] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: stopping tablet replica
I20260810 23:38:53.804530 6986 raft_consensus.cc:2229] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:53.804596 6992 raft_consensus.cc:2258] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:53.805100 6986 raft_consensus.cc:2258] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:53.805846 6993 ts_tablet_manager.cc:1918] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:53.806056 6993 log.cc:1196] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/42aec42f098546efbdf8a11b0b1dd375
I20260810 23:38:53.806622 6993 ts_tablet_manager.cc:1939] T 42aec42f098546efbdf8a11b0b1dd375 P 9b364bc98b1d447a9ab5fc069a668d2a: Deleting consensus metadata
I20260810 23:38:53.807089 6986 ts_tablet_manager.cc:1905] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:53.807428 6992 ts_tablet_manager.cc:1905] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:53.808650 6035 catalog_manager.cc:4841] TS 9b364bc98b1d447a9ab5fc069a668d2a (127.30.33.1:38001): tablet 42aec42f098546efbdf8a11b0b1dd375 (table test-table [id=e881edadc68a4b8ba58cb77546631b0a]) successfully deleted
I20260810 23:38:53.814064 6986 ts_tablet_manager.cc:1918] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:53.814293 6986 log.cc:1196] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/42aec42f098546efbdf8a11b0b1dd375
I20260810 23:38:53.814831 6986 ts_tablet_manager.cc:1939] T 42aec42f098546efbdf8a11b0b1dd375 P 5b186ba15bf345a2a8c0d28d0bec457a: Deleting consensus metadata
I20260810 23:38:53.815011 6992 ts_tablet_manager.cc:1918] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:53.815205 6992 log.cc:1196] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/42aec42f098546efbdf8a11b0b1dd375
I20260810 23:38:53.815907 6992 ts_tablet_manager.cc:1939] T 42aec42f098546efbdf8a11b0b1dd375 P ff75ad128e03452a81bb1d8b42e8ba4e: Deleting consensus metadata
I20260810 23:38:53.816977 6039 catalog_manager.cc:4841] TS 5b186ba15bf345a2a8c0d28d0bec457a (127.30.33.2:38003): tablet 42aec42f098546efbdf8a11b0b1dd375 (table test-table [id=e881edadc68a4b8ba58cb77546631b0a]) successfully deleted
I20260810 23:38:53.817775 6036 catalog_manager.cc:4841] TS ff75ad128e03452a81bb1d8b42e8ba4e (127.30.33.3:42879): tablet 42aec42f098546efbdf8a11b0b1dd375 (table test-table [id=e881edadc68a4b8ba58cb77546631b0a]) successfully deleted
Aug 10 23:38:53 dist-test-slave-pww0 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:38:53.826310 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.792592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.30.33.1:38001)
0810 23:38:53.792779 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.792785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.807415 (+ 14630us) server_negotiation.cc:407] Connection header received
0810 23:38:53.807986 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.807992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.808087 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.808160 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:53.809931 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.811146 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.813029 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.813694 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.813801 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.819111 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:53.819137 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:53.819160 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:53.819221 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:53.823062 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.823952 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.823963 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.823967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.824170 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.824582 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.824588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.824592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.825107 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:53.825467 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.825655 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.825915 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:38:53.827138 6402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:53.918020 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6185
I20260810 23:38:53.949749 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6421
Aug 10 23:38:53 dist-test-slave-pww0 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:38:53.968295 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:53.950614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.30.33.2:38003)
0810 23:38:53.950781 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:53.950785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:53.950804 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:38:53.951016 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:53.951021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:53.951119 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:53.951199 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:53.952905 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.953631 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.955543 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:53.955984 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:53.956117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:53.961534 (+ 5417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:53.961555 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:53.961577 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:53.961632 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:53.964734 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.965810 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.965821 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.965823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.966051 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:53.966546 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:53.966553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:53.966557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:53.967041 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:53.967318 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:53.967758 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 23:38:53.967965 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:38:53.969053 6620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:54.061084 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6421
I20260810 23:38:54.096766 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6638
Aug 10 23:38:54 dist-test-slave-pww0 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:38:54.113428 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:54.097621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.30.33.3:42879)
0810 23:38:54.097812 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:54.097818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:54.097926 (+ 108us) server_negotiation.cc:407] Connection header received
0810 23:38:54.098134 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:54.098139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:54.098210 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:54.098304 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:54.099687 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:54.100512 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:54.101755 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:54.102220 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:54.102337 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:54.106990 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:54.107013 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:54.107029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:54.107076 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:54.110408 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:54.111143 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:54.111152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:54.111154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:54.111322 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:54.111768 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:54.111772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:54.111774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:54.112178 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:54.112508 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:54.112829 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 23:38:54.113046 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:38:54.114253 6893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:54.238082 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6638
I20260810 23:38:54.257663 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6013
I20260810 23:38:54.258915 6125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:54.370481 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6013
2026-08-10T23:38:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4943 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T23:38:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:54Z Disabled control of system clock
I20260810 23:38:54.460160 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35779
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:38:54.811784 7007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:54.812110 7007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:54.812209 7007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:54.819818 7007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:54.819950 7007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:54.819988 7007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:54.820045 7007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:54.820091 7007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:54.829717 7007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:35779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:54.831128 7007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:54.833009 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:54.845755 7017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:54.845755 7019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:54.845791 7016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:54.846578 7007 server_base.cc:1034] running on GCE node
I20260810 23:38:54.847435 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:54.849088 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:54.850312 7007 hybrid_clock.cc:648] HybridClock initialized: now 1786405134850269 us; error 45 us; skew 500 ppm
I20260810 23:38:54.850669 7007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:54.853016 7007 webserver.cc:460] Webserver started at http://127.30.33.62:42505/ using document root <none> and password file <none>
I20260810 23:38:54.853753 7007 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:54.853864 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:54.854203 7007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:54.860010 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "e4762dce4b014bb6ab8b32978837a7e7"
format_stamp: "Formatted at 2026-08-10 23:38:54 on dist-test-slave-pww0"
server_key: "09377330a5f15b3c3c4ac174a2691f1b"
server_key_iv: "4991d6f811a4ce98a807835faa18ed32"
server_key_version: "encryptionkey@0"
I20260810 23:38:54.860860 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "e4762dce4b014bb6ab8b32978837a7e7"
format_stamp: "Formatted at 2026-08-10 23:38:54 on dist-test-slave-pww0"
server_key: "09377330a5f15b3c3c4ac174a2691f1b"
server_key_iv: "4991d6f811a4ce98a807835faa18ed32"
server_key_version: "encryptionkey@0"
I20260810 23:38:54.866333 7007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:38:54.870074 7028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:54.871613 7007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:38:54.871775 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "e4762dce4b014bb6ab8b32978837a7e7"
format_stamp: "Formatted at 2026-08-10 23:38:54 on dist-test-slave-pww0"
server_key: "09377330a5f15b3c3c4ac174a2691f1b"
server_key_iv: "4991d6f811a4ce98a807835faa18ed32"
server_key_version: "encryptionkey@0"
I20260810 23:38:54.871974 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:54.877970 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:54.878808 7007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:54.879089 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:54.897280 7007 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:35779
I20260810 23:38:54.897316 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:35779 every 8 connection(s)
I20260810 23:38:54.898725 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:54.902930 7126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:54.905857 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7007
I20260810 23:38:54.906100 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:54.906987 30852 external_mini_cluster.cc:1428] Setting key 231d591a8fdb71161660eb5e88433531
I20260810 23:38:54.916694 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:54.909653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.30.33.62:35779)
0810 23:38:54.910828 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:54.910841 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:54.910888 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:38:54.912690 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:54.912717 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:54.913217 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:54.913670 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:54.914868 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:54.914875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:54.914891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:54.914906 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:54.914979 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:54.915008 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:54.915241 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:54.915384 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 23:38:54.915723 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":430,"threads_started":1}
I20260810 23:38:54.917060 7126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7: Bootstrap starting.
I20260810 23:38:54.920845 7126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:54.922030 7126 log.cc:824] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:54.925442 7126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7: No bootstrap required, opened a new log
I20260810 23:38:54.933533 7126 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4762dce4b014bb6ab8b32978837a7e7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35779 } }
I20260810 23:38:54.933991 7126 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:54.934103 7126 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4762dce4b014bb6ab8b32978837a7e7, State: Initialized, Role: FOLLOWER
I20260810 23:38:54.934743 7126 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4762dce4b014bb6ab8b32978837a7e7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35779 } }
I20260810 23:38:54.935020 7126 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:54.935199 7126 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:54.935389 7126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:54.937669 7126 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4762dce4b014bb6ab8b32978837a7e7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35779 } }
I20260810 23:38:54.938158 7126 leader_election.cc:302] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4762dce4b014bb6ab8b32978837a7e7; no voters:
I20260810 23:38:54.938721 7126 leader_election.cc:288] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:54.938886 7133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:54.939654 7133 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [term 1 LEADER]: Becoming Leader. State: Replica: e4762dce4b014bb6ab8b32978837a7e7, State: Running, Role: LEADER
I20260810 23:38:54.940243 7133 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4762dce4b014bb6ab8b32978837a7e7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35779 } }
I20260810 23:38:54.940941 7126 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:54.943048 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4762dce4b014bb6ab8b32978837a7e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4762dce4b014bb6ab8b32978837a7e7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35779 } } }
I20260810 23:38:54.943617 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:54.943616 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4762dce4b014bb6ab8b32978837a7e7. Latest consensus state: current_term: 1 leader_uuid: "e4762dce4b014bb6ab8b32978837a7e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4762dce4b014bb6ab8b32978837a7e7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35779 } } }
I20260810 23:38:54.943984 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:54.944862 7143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:54.951305 7143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:54.960824 7143 catalog_manager.cc:1347] Generated new cluster ID: 7473808e3f544247b52ae833762f3f66
I20260810 23:38:54.960959 7143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:54.994171 7143 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:54.994750 7143 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:55.004688 7143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7: Generated new TSK 0
I20260810 23:38:55.005676 7143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:38:55.018509 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35779
--builtin_ntp_servers=127.30.33.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:38:55.373924 7172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:55.374239 7172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:55.374378 7172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:55.382009 7172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:55.382140 7172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:55.382310 7172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:38:55.391834 7172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:55.393270 7172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:55.395191 7172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:55.408694 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:55.408811 7184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:55.408617 7186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:55.409883 7172 server_base.cc:1034] running on GCE node
I20260810 23:38:55.410588 7172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:55.411648 7172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:55.412894 7172 hybrid_clock.cc:648] HybridClock initialized: now 1786405135412824 us; error 55 us; skew 500 ppm
I20260810 23:38:55.413271 7172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:55.415726 7172 webserver.cc:460] Webserver started at http://127.30.33.1:46573/ using document root <none> and password file <none>
I20260810 23:38:55.416560 7172 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:55.416661 7172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:55.416965 7172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:55.420228 7172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "5d280fd411e44b4d8eeed14023868ceb"
format_stamp: "Formatted at 2026-08-10 23:38:55 on dist-test-slave-pww0"
server_key: "fdb85de98f4b89bf144fd2e590b9d6b2"
server_key_iv: "3c8e7369d2e056d8d5e5639aa59ecfc9"
server_key_version: "encryptionkey@0"
I20260810 23:38:55.421139 7172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "5d280fd411e44b4d8eeed14023868ceb"
format_stamp: "Formatted at 2026-08-10 23:38:55 on dist-test-slave-pww0"
server_key: "fdb85de98f4b89bf144fd2e590b9d6b2"
server_key_iv: "3c8e7369d2e056d8d5e5639aa59ecfc9"
server_key_version: "encryptionkey@0"
I20260810 23:38:55.426990 7172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:38:55.431218 7198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:55.432951 7172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:55.433137 7172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "5d280fd411e44b4d8eeed14023868ceb"
format_stamp: "Formatted at 2026-08-10 23:38:55 on dist-test-slave-pww0"
server_key: "fdb85de98f4b89bf144fd2e590b9d6b2"
server_key_iv: "3c8e7369d2e056d8d5e5639aa59ecfc9"
server_key_version: "encryptionkey@0"
I20260810 23:38:55.433395 7172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:55.439666 7172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:55.440555 7172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:55.440821 7172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:55.441890 7172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:55.443516 7172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:55.443622 7172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:55.443732 7172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:55.443810 7172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:55.476215 7172 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:35731
I20260810 23:38:55.476253 7422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:35731 every 8 connection(s)
I20260810 23:38:55.477638 7172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:38:55.484491 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7172
I20260810 23:38:55.484719 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:38:55.485392 30852 external_mini_cluster.cc:1428] Setting key d79277c3a561a3953e65f8cfba93fc98
I20260810 23:38:55.486596 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:55.480406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:38471 (local address 127.30.33.62:35779)
0810 23:38:55.480885 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:55.480890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:55.482201 (+ 1311us) server_negotiation.cc:407] Connection header received
0810 23:38:55.484177 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:55.484183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:55.484261 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:55.484398 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:55.485780 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:55.485786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:55.485791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:55.485814 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:55.485869 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:55.485886 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:55.486025 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:55.486199 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:38:55.486269 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260810 23:38:55.487850 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:55.481067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35779 (local address 127.30.33.1:38471)
0810 23:38:55.482032 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:55.482070 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:55.483857 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:55.484589 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:55.484597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:55.484724 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:55.485419 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:55.485422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:55.485484 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:55.485595 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:55.485986 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:55.485995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:55.486003 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:38:55.486300 (+ 297us) client_negotiation.cc:240] Negotiation successful
0810 23:38:55.486558 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260810 23:38:55.488583 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35779
--builtin_ntp_servers=127.30.33.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:38:55.489487 7430 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35779
I20260810 23:38:55.489850 7430 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:55.490651 7430 heartbeater.cc:511] Master 127.30.33.62:35779 requested a full tablet report, sending...
I20260810 23:38:55.493031 7051 ts_manager.cc:194] Registered new tserver with Master: 5d280fd411e44b4d8eeed14023868ceb (127.30.33.1:35731)
W20260810 23:38:55.837843 7437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:55.838204 7437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:55.838330 7437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:55.846277 7437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:55.846403 7437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:55.846529 7437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:38:55.856285 7437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:55.857739 7437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:55.859658 7437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:55.873597 7451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:55.873646 7455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:55.873571 7449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:55.876338 7437 server_base.cc:1034] running on GCE node
I20260810 23:38:55.877130 7437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:55.878283 7437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:55.879550 7437 hybrid_clock.cc:648] HybridClock initialized: now 1786405135879477 us; error 53 us; skew 500 ppm
I20260810 23:38:55.879900 7437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:55.882468 7437 webserver.cc:460] Webserver started at http://127.30.33.2:38947/ using document root <none> and password file <none>
I20260810 23:38:55.883250 7437 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:55.883374 7437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:55.883862 7437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:55.887246 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "3d0e7fd56956416daa8bc7535296942f"
format_stamp: "Formatted at 2026-08-10 23:38:55 on dist-test-slave-pww0"
server_key: "672b0869e965d69b73c08ffd6881fec6"
server_key_iv: "4075b0f4ee5b0bed1c7ac9ebfffcdcbe"
server_key_version: "encryptionkey@0"
I20260810 23:38:55.888229 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "3d0e7fd56956416daa8bc7535296942f"
format_stamp: "Formatted at 2026-08-10 23:38:55 on dist-test-slave-pww0"
server_key: "672b0869e965d69b73c08ffd6881fec6"
server_key_iv: "4075b0f4ee5b0bed1c7ac9ebfffcdcbe"
server_key_version: "encryptionkey@0"
I20260810 23:38:55.894243 7437 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:38:55.898445 7472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:55.900082 7437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:55.900264 7437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "3d0e7fd56956416daa8bc7535296942f"
format_stamp: "Formatted at 2026-08-10 23:38:55 on dist-test-slave-pww0"
server_key: "672b0869e965d69b73c08ffd6881fec6"
server_key_iv: "4075b0f4ee5b0bed1c7ac9ebfffcdcbe"
server_key_version: "encryptionkey@0"
I20260810 23:38:55.900502 7437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:55.906899 7437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:55.907863 7437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:55.908142 7437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:55.909227 7437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:55.910880 7437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:55.910982 7437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:55.911104 7437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:55.911172 7437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:55.945921 7437 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:43781
I20260810 23:38:55.945983 7685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:43781 every 8 connection(s)
I20260810 23:38:55.947355 7437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:38:55.956128 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:55.950056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:57123 (local address 127.30.33.62:35779)
0810 23:38:55.950309 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:55.950316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:55.951944 (+ 1628us) server_negotiation.cc:407] Connection header received
0810 23:38:55.953742 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:55.953747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:55.953818 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:55.953927 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:55.955372 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:55.955378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:55.955381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:55.955403 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:55.955443 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:55.955492 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:55.955636 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:55.955817 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 23:38:55.955880 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:38:55.957343 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:55.950690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35779 (local address 127.30.33.2:57123)
0810 23:38:55.951757 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:55.951799 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:55.953414 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:55.954159 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:55.954168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:55.954278 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:55.954994 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:55.954996 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:55.955042 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:55.955147 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:55.955589 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:55.955599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:55.955606 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:38:55.955883 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 23:38:55.956136 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":302,"threads_started":1}
I20260810 23:38:55.957372 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7437
I20260810 23:38:55.957510 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:38:55.958006 30852 external_mini_cluster.cc:1428] Setting key 4d012243c34ffcb159eaa5d742abd4ec
I20260810 23:38:55.959082 7686 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35779
I20260810 23:38:55.959481 7686 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:55.960345 7686 heartbeater.cc:511] Master 127.30.33.62:35779 requested a full tablet report, sending...
I20260810 23:38:55.961488 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35779
--builtin_ntp_servers=127.30.33.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:38:55.962204 7051 ts_manager.cc:194] Registered new tserver with Master: 3d0e7fd56956416daa8bc7535296942f (127.30.33.2:43781)
W20260810 23:38:56.305646 7691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:56.305987 7691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:56.306102 7691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:56.313473 7691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:56.313593 7691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:56.313704 7691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:38:56.323027 7691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:56.324438 7691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:56.326241 7691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:56.340834 7701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:56.340838 7700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:56.341370 7704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:56.341748 7691 server_base.cc:1034] running on GCE node
I20260810 23:38:56.342470 7691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:56.343576 7691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:56.344780 7691 hybrid_clock.cc:648] HybridClock initialized: now 1786405136344710 us; error 61 us; skew 500 ppm
I20260810 23:38:56.345139 7691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:56.347723 7691 webserver.cc:460] Webserver started at http://127.30.33.3:39481/ using document root <none> and password file <none>
I20260810 23:38:56.348491 7691 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:56.348590 7691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:56.348930 7691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:56.352030 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "a2f50c64d5bf4c79a81970bb4c9f514d"
format_stamp: "Formatted at 2026-08-10 23:38:56 on dist-test-slave-pww0"
server_key: "56c05ed434690ffe7ff73cc8a3054e7a"
server_key_iv: "b24d90c8b3ea24688f2b72c4df93281e"
server_key_version: "encryptionkey@0"
I20260810 23:38:56.352882 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "a2f50c64d5bf4c79a81970bb4c9f514d"
format_stamp: "Formatted at 2026-08-10 23:38:56 on dist-test-slave-pww0"
server_key: "56c05ed434690ffe7ff73cc8a3054e7a"
server_key_iv: "b24d90c8b3ea24688f2b72c4df93281e"
server_key_version: "encryptionkey@0"
I20260810 23:38:56.358707 7691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:38:56.362666 7714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:56.364321 7691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:56.364518 7691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "a2f50c64d5bf4c79a81970bb4c9f514d"
format_stamp: "Formatted at 2026-08-10 23:38:56 on dist-test-slave-pww0"
server_key: "56c05ed434690ffe7ff73cc8a3054e7a"
server_key_iv: "b24d90c8b3ea24688f2b72c4df93281e"
server_key_version: "encryptionkey@0"
I20260810 23:38:56.364770 7691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:56.371513 7691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:56.372427 7691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:56.372696 7691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:56.373680 7691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:38:56.375227 7691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:38:56.375337 7691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:56.375483 7691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:38:56.375569 7691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 23:38:56.407035 7691 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:45149
I20260810 23:38:56.407087 7921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:45149 every 8 connection(s)
I20260810 23:38:56.408525 7691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:38:56.418110 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7691
I20260810 23:38:56.418256 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:38:56.418716 30852 external_mini_cluster.cc:1428] Setting key 7cea74fe1e4325d455dd16e2892f6450
I20260810 23:38:56.420447 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.411354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:32861 (local address 127.30.33.62:35779)
0810 23:38:56.411598 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.411605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.413095 (+ 1490us) server_negotiation.cc:407] Connection header received
0810 23:38:56.414778 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.414782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.414839 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.414931 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.419588 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.419595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.419599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.419624 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.419671 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.419688 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.419828 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.420085 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.420166 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:38:56.421919 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.411911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35779 (local address 127.30.33.3:32861)
0810 23:38:56.412923 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.412966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.414519 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.415139 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.415147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.415248 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.419151 (+ 3903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.419154 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.419206 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.419326 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.419817 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.419830 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.419839 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.420183 (+ 344us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.420500 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":310,"threads_started":1}
I20260810 23:38:56.423487 7924 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35779
I20260810 23:38:56.423839 7924 heartbeater.cc:464] Registering TS with master...
I20260810 23:38:56.424492 7924 heartbeater.cc:511] Master 127.30.33.62:35779 requested a full tablet report, sending...
I20260810 23:38:56.425966 7051 ts_manager.cc:194] Registered new tserver with Master: a2f50c64d5bf4c79a81970bb4c9f514d (127.30.33.3:45149)
I20260810 23:38:56.435503 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:38:56.443145 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.440468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.30.33.62:35779)
0810 23:38:56.440725 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.440731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.441140 (+ 409us) server_negotiation.cc:407] Connection header received
0810 23:38:56.441329 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.441333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.441392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.441484 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.442451 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.442457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.442460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.442476 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.442510 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.442522 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.442638 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.442767 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.442832 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 23:38:56.447207 7051 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:38:56.454125 7051 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:38:56.474082 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.466664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:45149 (local address 127.0.0.1:53010)
0810 23:38:56.468708 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.468778 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.469097 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.471299 (+ 2202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.471309 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.471431 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.472390 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.472395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.472432 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.472546 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.473195 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.473210 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.473220 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.473428 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.473543 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1760,"thread_start_us":2141,"threads_started":1}
I20260810 23:38:56.474923 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.466880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.30.33.3:45149)
0810 23:38:56.468097 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.468107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.468955 (+ 848us) server_negotiation.cc:407] Connection header received
0810 23:38:56.469423 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.469434 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.469712 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.469873 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.472798 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.472810 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.472830 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.472854 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.472926 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.472950 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.473134 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.474346 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.474473 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":805,"threads_started":1}
I20260810 23:38:56.480798 7802 tablet_service.cc:1467] Processing CreateTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 (DEFAULT_TABLE table=test-table [id=5bdce4027fc74a66b2db5f205965f47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:56.482833 7802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca35da0ae77c4168a63fdeb9a1909b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:56.483273 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.467253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:43781 (local address 127.0.0.1:48088)
0810 23:38:56.478602 (+ 11349us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.478623 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.478751 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.481063 (+ 2312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.481069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.481097 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.481838 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.481841 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.481861 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.481920 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.482541 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.482547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.482552 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.482703 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.482767 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11188,"threads_started":1}
I20260810 23:38:56.484079 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.468210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.30.33.2:43781)
0810 23:38:56.479261 (+ 11051us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.479271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.479307 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:38:56.479379 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.479387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.479665 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.479841 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.482188 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.482198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.482215 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.482245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.482316 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.482338 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.482475 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.483717 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.483836 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10848,"thread_start_us":12571,"threads_started":1}
I20260810 23:38:56.487247 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.467011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43748)
0810 23:38:56.475531 (+ 8520us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.475554 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.483037 (+ 7483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.484280 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.484285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.484325 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.484925 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.484929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.484948 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.484999 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.486784 (+ 1785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.486789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.486794 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.486896 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.486957 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8364,"thread_start_us":14823,"threads_started":1}
I20260810 23:38:56.487378 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.467949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.30.33.1:35731)
0810 23:38:56.483667 (+ 15718us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.483676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.483707 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:38:56.483784 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.483793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.484056 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.484634 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.486562 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.486571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.486585 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.486602 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.486664 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.486683 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.486855 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.486923 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.487023 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15530,"thread_start_us":4140,"threads_started":1}
I20260810 23:38:56.488351 7549 tablet_service.cc:1467] Processing CreateTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 (DEFAULT_TABLE table=test-table [id=5bdce4027fc74a66b2db5f205965f47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:56.490201 7549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca35da0ae77c4168a63fdeb9a1909b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:56.491698 7284 tablet_service.cc:1467] Processing CreateTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 (DEFAULT_TABLE table=test-table [id=5bdce4027fc74a66b2db5f205965f47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:38:56.493278 7284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca35da0ae77c4168a63fdeb9a1909b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:56.496531 7430 heartbeater.cc:503] Master 127.30.33.62:35779 was elected leader, sending a full tablet report...
I20260810 23:38:56.500993 7960 tablet_bootstrap.cc:492] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Bootstrap starting.
I20260810 23:38:56.504272 7960 tablet_bootstrap.cc:654] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:56.505656 7960 log.cc:824] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:56.507997 7961 tablet_bootstrap.cc:492] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Bootstrap starting.
I20260810 23:38:56.509071 7966 tablet_bootstrap.cc:492] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: Bootstrap starting.
I20260810 23:38:56.510191 7960 tablet_bootstrap.cc:492] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: No bootstrap required, opened a new log
I20260810 23:38:56.510485 7960 ts_tablet_manager.cc:1397] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:38:56.512374 7961 tablet_bootstrap.cc:654] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:56.512885 7966 tablet_bootstrap.cc:654] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:56.513752 7961 log.cc:824] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:56.514150 7966 log.cc:824] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:56.518349 7966 tablet_bootstrap.cc:492] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: No bootstrap required, opened a new log
I20260810 23:38:56.518718 7966 ts_tablet_manager.cc:1397] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:38:56.519127 7961 tablet_bootstrap.cc:492] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: No bootstrap required, opened a new log
I20260810 23:38:56.518862 7960 raft_consensus.cc:348] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.519275 7960 raft_consensus.cc:374] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:56.519413 7960 raft_consensus.cc:729] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f50c64d5bf4c79a81970bb4c9f514d, State: Initialized, Role: FOLLOWER
I20260810 23:38:56.519451 7961 ts_tablet_manager.cc:1397] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260810 23:38:56.520072 7960 consensus_queue.cc:260] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.521219 7924 heartbeater.cc:503] Master 127.30.33.62:35779 was elected leader, sending a full tablet report...
I20260810 23:38:56.521976 7960 ts_tablet_manager.cc:1428] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260810 23:38:56.529812 7966 raft_consensus.cc:348] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.530447 7966 raft_consensus.cc:374] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:56.530575 7966 raft_consensus.cc:729] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d280fd411e44b4d8eeed14023868ceb, State: Initialized, Role: FOLLOWER
I20260810 23:38:56.531008 7961 raft_consensus.cc:348] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.531569 7961 raft_consensus.cc:374] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:56.531374 7966 consensus_queue.cc:260] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.531682 7961 raft_consensus.cc:729] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0e7fd56956416daa8bc7535296942f, State: Initialized, Role: FOLLOWER
I20260810 23:38:56.532454 7961 consensus_queue.cc:260] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.533604 7966 ts_tablet_manager.cc:1428] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260810 23:38:56.533692 7686 heartbeater.cc:503] Master 127.30.33.62:35779 was elected leader, sending a full tablet report...
I20260810 23:38:56.537057 7961 ts_tablet_manager.cc:1428] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Time spent starting tablet: real 0.017s user 0.006s sys 0.008s
I20260810 23:38:56.547554 7969 raft_consensus.cc:482] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:56.547930 7969 raft_consensus.cc:504] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.549741 7973 raft_consensus.cc:482] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:56.549929 7969 leader_election.cc:288] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d280fd411e44b4d8eeed14023868ceb (127.30.33.1:35731), 3d0e7fd56956416daa8bc7535296942f (127.30.33.2:43781)
I20260810 23:38:56.550676 7973 raft_consensus.cc:504] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.552750 7972 raft_consensus.cc:482] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:56.553169 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.550499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:42899 (local address 127.30.33.1:35731)
0810 23:38:56.550713 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.550719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.550896 (+ 177us) server_negotiation.cc:407] Connection header received
0810 23:38:56.551085 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.551091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.551165 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.551270 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.552438 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.552444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.552448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.552469 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.552512 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.552529 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.552658 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.552854 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.552926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:38:56.553200 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.550526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.30.33.3:42899)
0810 23:38:56.550808 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.550823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.550942 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.551497 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.551502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.551531 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.552229 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.552233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.552252 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.552297 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.552708 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.552713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.552719 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.552844 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.552905 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 23:38:56.553117 7972 raft_consensus.cc:504] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.554437 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.550519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:34243 (local address 127.30.33.2:43781)
0810 23:38:56.550711 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.550717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.551765 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 23:38:56.551941 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.551948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.552006 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.552107 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.553724 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.553731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.553735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.553756 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.553794 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.553810 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.553900 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.554098 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.554160 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:38:56.554298 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d280fd411e44b4d8eeed14023868ceb" is_pre_election: true
I20260810 23:38:56.554589 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.550902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:43781 (local address 127.30.33.3:34243)
0810 23:38:56.551640 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.551662 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.551781 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.552396 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.552399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.552423 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.553454 (+ 1031us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.553458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.553475 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.553525 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.553975 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.553978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.553981 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.554078 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.554132 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":317,"threads_started":1}
I20260810 23:38:56.554744 7335 raft_consensus.cc:2454] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f50c64d5bf4c79a81970bb4c9f514d in term 0.
I20260810 23:38:56.555264 7972 leader_election.cc:288] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f50c64d5bf4c79a81970bb4c9f514d (127.30.33.3:45149), 3d0e7fd56956416daa8bc7535296942f (127.30.33.2:43781)
I20260810 23:38:56.556116 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0e7fd56956416daa8bc7535296942f" is_pre_election: true
I20260810 23:38:56.556259 7721 leader_election.cc:302] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d280fd411e44b4d8eeed14023868ceb, a2f50c64d5bf4c79a81970bb4c9f514d; no voters:
I20260810 23:38:56.556619 7586 raft_consensus.cc:2454] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f50c64d5bf4c79a81970bb4c9f514d in term 0.
I20260810 23:38:56.557327 7973 leader_election.cc:288] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f50c64d5bf4c79a81970bb4c9f514d (127.30.33.3:45149), 5d280fd411e44b4d8eeed14023868ceb (127.30.33.1:35731)
I20260810 23:38:56.559258 7969 raft_consensus.cc:2781] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:38:56.560492 7969 raft_consensus.cc:482] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:38:56.561367 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.556818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:32983 (local address 127.30.33.2:43781)
0810 23:38:56.556959 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.556963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.558092 (+ 1129us) server_negotiation.cc:407] Connection header received
0810 23:38:56.558314 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.558320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.558401 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.558467 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.560184 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.560191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.560196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.560219 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.560258 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.560274 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.560371 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.560894 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.560969 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 23:38:56.562355 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.555617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:43781 (local address 127.30.33.1:32983)
0810 23:38:56.557971 (+ 2354us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.557994 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.558179 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.558676 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.558681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.558707 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.559854 (+ 1147us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.559857 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.559957 (+ 100us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.560004 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.560745 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.560750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.560755 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.561800 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.561873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"spinlock_wait_cycles":2432,"thread_start_us":2201,"threads_started":1}
I20260810 23:38:56.562637 7969 raft_consensus.cc:3037] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:56.564206 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.555897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:35277 (local address 127.30.33.3:45149)
0810 23:38:56.556065 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.556071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.556580 (+ 509us) server_negotiation.cc:407] Connection header received
0810 23:38:56.556840 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.556845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.556904 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.556973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.559789 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.559796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.559801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.559821 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.559857 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.559872 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.560002 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.563882 (+ 3880us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.563951 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:38:56.566373 7969 raft_consensus.cc:504] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.567260 7969 leader_election.cc:288] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [CANDIDATE]: Term 1 election: Requested vote from peers 5d280fd411e44b4d8eeed14023868ceb (127.30.33.1:35731), 3d0e7fd56956416daa8bc7535296942f (127.30.33.2:43781)
I20260810 23:38:56.565285 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.555400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:45149 (local address 127.30.33.1:35277)
0810 23:38:56.556482 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.556502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.556686 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.557181 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.557185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.557211 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.559578 (+ 2367us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.559582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.559603 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.559661 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.563525 (+ 3864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.563531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.563537 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.564815 (+ 1278us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.564876 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":767,"threads_started":1}
I20260810 23:38:56.568298 7586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0e7fd56956416daa8bc7535296942f"
I20260810 23:38:56.568593 7586 raft_consensus.cc:3037] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:56.568616 7841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "5d280fd411e44b4d8eeed14023868ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" is_pre_election: true
I20260810 23:38:56.568863 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d280fd411e44b4d8eeed14023868ceb"
I20260810 23:38:56.569119 7335 raft_consensus.cc:3037] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:56.569334 7841 raft_consensus.cc:2379] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d280fd411e44b4d8eeed14023868ceb in current term 1: Already voted for candidate a2f50c64d5bf4c79a81970bb4c9f514d in this term.
I20260810 23:38:56.571097 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.557752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:52827 (local address 127.30.33.3:45149)
0810 23:38:56.564446 (+ 6694us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.564452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.568476 (+ 4024us) server_negotiation.cc:407] Connection header received
0810 23:38:56.568723 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.568729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.568850 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.568930 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.570306 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.570311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.570316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.570337 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.570376 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.570391 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.570519 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.570728 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.570795 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6584,"thread_start_us":10514,"threads_started":1}
I20260810 23:38:56.571873 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.557398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:45149 (local address 127.30.33.2:52827)
0810 23:38:56.568374 (+ 10976us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.568401 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.568570 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.569168 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.569173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.569199 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.570012 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.570015 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.570037 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.570103 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.570583 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.570589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.570594 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.571334 (+ 740us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.571393 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10822,"thread_start_us":10628,"threads_started":1}
I20260810 23:38:56.571702 7583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "5d280fd411e44b4d8eeed14023868ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0e7fd56956416daa8bc7535296942f" is_pre_election: true
I20260810 23:38:56.572764 7335 raft_consensus.cc:2454] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f50c64d5bf4c79a81970bb4c9f514d in term 1.
I20260810 23:38:56.572868 7202 leader_election.cc:302] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d280fd411e44b4d8eeed14023868ceb; no voters: 3d0e7fd56956416daa8bc7535296942f, a2f50c64d5bf4c79a81970bb4c9f514d
I20260810 23:38:56.573124 7841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "3d0e7fd56956416daa8bc7535296942f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" is_pre_election: true
I20260810 23:38:56.573377 7841 raft_consensus.cc:2379] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d0e7fd56956416daa8bc7535296942f in current term 1: Already voted for candidate a2f50c64d5bf4c79a81970bb4c9f514d in this term.
I20260810 23:38:56.573642 7972 raft_consensus.cc:2726] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:56.574038 7721 leader_election.cc:302] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d280fd411e44b4d8eeed14023868ceb, a2f50c64d5bf4c79a81970bb4c9f514d; no voters:
I20260810 23:38:56.574263 7586 raft_consensus.cc:2454] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f50c64d5bf4c79a81970bb4c9f514d in term 1.
I20260810 23:38:56.574712 7969 raft_consensus.cc:2781] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:56.574806 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.557974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.30.33.2:44055)
0810 23:38:56.572104 (+ 14130us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.572128 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:56.572228 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:56.573258 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:56.573262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:56.573299 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:56.573946 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:38:56.573949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:56.573966 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:38:56.574012 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:56.574364 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:56.574369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:56.574374 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:38:56.574493 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:38:56.574549 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14023,"threads_started":1}
I20260810 23:38:56.574826 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.558584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:44055 (local address 127.30.33.1:35731)
0810 23:38:56.558749 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.558755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.572943 (+ 14188us) server_negotiation.cc:407] Connection header received
0810 23:38:56.573012 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.573017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.573118 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.573222 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.574160 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.574166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.574171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.574190 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.574229 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.574256 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.574391 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.574487 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.574551 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:38:56.575402 7969 raft_consensus.cc:686] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 1 LEADER]: Becoming Leader. State: Replica: a2f50c64d5bf4c79a81970bb4c9f514d, State: Running, Role: LEADER
I20260810 23:38:56.575994 7969 consensus_queue.cc:237] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } }
I20260810 23:38:56.579280 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35da0ae77c4168a63fdeb9a1909b41" candidate_uuid: "3d0e7fd56956416daa8bc7535296942f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d280fd411e44b4d8eeed14023868ceb" is_pre_election: true
I20260810 23:38:56.579653 7335 raft_consensus.cc:2379] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d0e7fd56956416daa8bc7535296942f in current term 1: Already voted for candidate a2f50c64d5bf4c79a81970bb4c9f514d in this term.
I20260810 23:38:56.580364 7479 leader_election.cc:302] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0e7fd56956416daa8bc7535296942f; no voters: 5d280fd411e44b4d8eeed14023868ceb, a2f50c64d5bf4c79a81970bb4c9f514d
I20260810 23:38:56.580809 7973 raft_consensus.cc:2726] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:56.581091 7051 catalog_manager.cc:5495] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f50c64d5bf4c79a81970bb4c9f514d (127.30.33.3). New cstate: current_term: 1 leader_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f50c64d5bf4c79a81970bb4c9f514d" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 45149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 } health_report { overall_health: UNKNOWN } } }
I20260810 23:38:56.594158 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.592006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.30.33.3:45149)
0810 23:38:56.592196 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.592201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.592221 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:38:56.592433 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.592438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.592526 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.592613 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.593527 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.593532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.593537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.593554 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.593608 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.593621 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.593725 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.593841 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.593905 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:38:56.602841 7586 raft_consensus.cc:1261] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 1 FOLLOWER]: Refusing update from remote peer a2f50c64d5bf4c79a81970bb4c9f514d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:56.602861 7335 raft_consensus.cc:1261] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 1 FOLLOWER]: Refusing update from remote peer a2f50c64d5bf4c79a81970bb4c9f514d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:56.603754 7985 consensus_queue.cc:1035] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d280fd411e44b4d8eeed14023868ceb" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:56.604174 7969 consensus_queue.cc:1035] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0e7fd56956416daa8bc7535296942f" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:56.633776 7050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:56.634250 7050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:38:56.641685 7802 tablet_service.cc:1514] Processing DeleteTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:56 UTC) from {username='slave'} at 127.0.0.1:53010
I20260810 23:38:56.642027 7050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4762dce4b014bb6ab8b32978837a7e7: Sending DeleteTablet for 3 replicas of tablet ca35da0ae77c4168a63fdeb9a1909b41
I20260810 23:38:56.642803 7549 tablet_service.cc:1514] Processing DeleteTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:56 UTC) from {username='slave'} at 127.0.0.1:48088
I20260810 23:38:56.644109 7284 tablet_service.cc:1514] Processing DeleteTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:56 UTC) from {username='slave'} at 127.0.0.1:43748
I20260810 23:38:56.644464 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:56 UTC) from {username='slave'} at 127.0.0.1:48088
I20260810 23:38:56.644773 8016 tablet_replica.cc:330] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: stopping tablet replica
I20260810 23:38:56.645186 8016 raft_consensus.cc:2229] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:38:56.645819 8016 raft_consensus.cc:2258] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:56.646474 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7172
I20260810 23:38:56.648339 8016 ts_tablet_manager.cc:1905] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:56.651669 7797 tablet_service.cc:1514] Processing DeleteTablet for tablet ca35da0ae77c4168a63fdeb9a1909b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:38:56 UTC) from {username='slave'} at 127.0.0.1:53010
I20260810 23:38:56.652494 8015 tablet_replica.cc:330] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: stopping tablet replica
I20260810 23:38:56.653003 8015 raft_consensus.cc:2229] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:38:56.653716 8015 raft_consensus.cc:2258] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:56.654786 8016 ts_tablet_manager.cc:1918] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:56.655021 8016 log.cc:1196] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/ca35da0ae77c4168a63fdeb9a1909b41
I20260810 23:38:56.655706 8016 ts_tablet_manager.cc:1939] T ca35da0ae77c4168a63fdeb9a1909b41 P 3d0e7fd56956416daa8bc7535296942f: Deleting consensus metadata
I20260810 23:38:56.656843 8015 ts_tablet_manager.cc:1905] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:56.657508 8017 tablet_replica.cc:330] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: stopping tablet replica
I20260810 23:38:56.658386 8017 raft_consensus.cc:2229] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 23:38:56.658413 7034 catalog_manager.cc:4808] TS 3d0e7fd56956416daa8bc7535296942f (127.30.33.2:43781): delete failed for tablet ca35da0ae77c4168a63fdeb9a1909b41 because the tablet was not found. No further retry: Not found: Tablet not found: ca35da0ae77c4168a63fdeb9a1909b41
I20260810 23:38:56.659297 8017 raft_consensus.cc:2258] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:38:56.661206 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.648047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.30.33.1:35731)
0810 23:38:56.648257 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.648265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.657977 (+ 9712us) server_negotiation.cc:407] Connection header received
0810 23:38:56.658860 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.658866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.658967 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.659080 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.660389 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.660396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.660400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.660422 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.660462 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.660477 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.660565 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.660832 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.660903 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:38:56.662228 8017 ts_tablet_manager.cc:1905] T ca35da0ae77c4168a63fdeb9a1909b41 P 5d280fd411e44b4d8eeed14023868ceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:38:56.662461 7412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:56.664813 8015 ts_tablet_manager.cc:1918] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:38:56.665089 8015 log.cc:1196] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/ca35da0ae77c4168a63fdeb9a1909b41
I20260810 23:38:56.665761 8015 ts_tablet_manager.cc:1939] T ca35da0ae77c4168a63fdeb9a1909b41 P a2f50c64d5bf4c79a81970bb4c9f514d: Deleting consensus metadata
W20260810 23:38:56.668306 7032 catalog_manager.cc:4808] TS a2f50c64d5bf4c79a81970bb4c9f514d (127.30.33.3:45149): delete failed for tablet ca35da0ae77c4168a63fdeb9a1909b41 because the tablet was not found. No further retry: Not found: Tablet not found: ca35da0ae77c4168a63fdeb9a1909b41
I20260810 23:38:56.757414 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7172
I20260810 23:38:56.781731 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7437
W20260810 23:38:56.781944 7037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.33.1:35731 (error 108)
I20260810 23:38:56.783849 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.783383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43764)
0810 23:38:56.783559 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.783685 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:38:56.784173 7037 catalog_manager.cc:4584] TS 5d280fd411e44b4d8eeed14023868ceb (127.30.33.1:35731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca35da0ae77c4168a63fdeb9a1909b41: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
I20260810 23:38:56.785580 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.782852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.30.33.2:43781)
0810 23:38:56.783042 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.783048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.783356 (+ 308us) server_negotiation.cc:407] Connection header received
0810 23:38:56.783617 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.783624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.783723 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.783811 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.784987 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.784992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.784996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.785012 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.785045 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.785057 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.785168 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.785282 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.785333 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:38:56.786322 7677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:56.838100 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.837450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43770)
0810 23:38:56.837788 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.837913 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 23:38:56.839359 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.838920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43784)
0810 23:38:56.839086 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.839182 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:38:56.880754 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7437
I20260810 23:38:56.900527 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.899946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43798)
0810 23:38:56.900237 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.900344 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 23:38:56.902014 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.901506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43814)
0810 23:38:56.901751 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.901837 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 23:38:56.928640 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7691
I20260810 23:38:56.931953 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.929560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.30.33.3:45149)
0810 23:38:56.929764 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:56.929769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:56.929792 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:38:56.930044 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:56.930048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:56.930115 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:56.930199 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:56.931213 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:56.931218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:56.931221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:56.931238 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:56.931273 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:56.931285 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:56.931397 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:56.931575 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 23:38:56.931627 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:38:56.932726 7914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:56.982954 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.982387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43826)
0810 23:38:56.982672 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.982789 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 23:38:56.984216 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:56.983718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43834)
0810 23:38:56.983941 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:56.984058 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 23:38:57.056488 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7691
I20260810 23:38:57.085846 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7007
I20260810 23:38:57.087088 7116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:57.200474 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:57.199845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:35731 (local address 127.0.0.1:43842)
0810 23:38:57.200078 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:57.200234 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 23:38:57.200677 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7007
2026-08-10T23:38:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2824 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 23:38:57.248569 30852 mini_oidc.cc:152] Starting JWKS server
I20260810 23:38:57.251600 30852 webserver.cc:460] Webserver started at https://127.0.0.1:34801/ using document root <none> and password file <none>
I20260810 23:38:57.251988 30852 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:38:57.252663 30852 webserver.cc:460] Webserver started at http://127.0.0.1:39237/ using document root <none> and password file <none>
2026-08-10T23:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:57Z Disabled control of system clock
I20260810 23:38:57.289860 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46343
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:46343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34801/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/testchainca.cert with env {}
W20260810 23:38:57.634074 8047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:57.634430 8047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:57.634570 8047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:57.642212 8047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:57.642369 8047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:57.642438 8047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:57.642534 8047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:57.642596 8047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:38:57.642684 8047 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:38:57.642746 8047 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34801/jwks.json
I20260810 23:38:57.653481 8047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:46343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34801/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:57.654929 8047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:57.656385 8047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:57.669118 8053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:57.669059 8052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:57.669140 8059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:57.673244 8047 server_base.cc:1034] running on GCE node
I20260810 23:38:57.674194 8047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:57.675896 8047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:57.677120 8047 hybrid_clock.cc:648] HybridClock initialized: now 1786405137677067 us; error 53 us; skew 500 ppm
I20260810 23:38:57.677507 8047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:57.680116 8047 webserver.cc:460] Webserver started at http://127.30.33.62:44369/ using document root <none> and password file <none>
I20260810 23:38:57.680884 8047 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:57.681010 8047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:57.681351 8047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:57.684717 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6"
format_stamp: "Formatted at 2026-08-10 23:38:57 on dist-test-slave-pww0"
server_key: "45a381caace17509efa31b4922899845"
server_key_iv: "89432421ef0b2819eb745d9e2c377c77"
server_key_version: "encryptionkey@0"
I20260810 23:38:57.685635 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6"
format_stamp: "Formatted at 2026-08-10 23:38:57 on dist-test-slave-pww0"
server_key: "45a381caace17509efa31b4922899845"
server_key_iv: "89432421ef0b2819eb745d9e2c377c77"
server_key_version: "encryptionkey@0"
I20260810 23:38:57.691586 8047 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:38:57.697561 8068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:57.699163 8047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:38:57.699424 8047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6"
format_stamp: "Formatted at 2026-08-10 23:38:57 on dist-test-slave-pww0"
server_key: "45a381caace17509efa31b4922899845"
server_key_iv: "89432421ef0b2819eb745d9e2c377c77"
server_key_version: "encryptionkey@0"
I20260810 23:38:57.699714 8047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:57.714144 8047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:57.715097 8047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:57.715427 8047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:57.734408 8047 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:46343
I20260810 23:38:57.734445 8163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:46343 every 8 connection(s)
I20260810 23:38:57.735893 8047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:57.740108 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:57.745302 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8047
I20260810 23:38:57.745554 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:57.746379 30852 external_mini_cluster.cc:1428] Setting key 6f89abe086cb5f23c589316308a3b26f
I20260810 23:38:57.752434 8164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6: Bootstrap starting.
I20260810 23:38:57.756287 8164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:57.757551 8164 log.cc:824] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:57.761006 8164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6: No bootstrap required, opened a new log
I20260810 23:38:57.761528 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:57.748482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.30.33.62:46343)
0810 23:38:57.749672 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:57.749701 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:57.749768 (+ 67us) server_negotiation.cc:407] Connection header received
0810 23:38:57.751681 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:57.751708 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:57.752274 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:57.752671 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:57.753319 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:57.754697 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:57.756643 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:57.757399 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:57.757545 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:57.758961 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:57.758972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:57.758995 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:57.759653 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:57.759761 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:57.759786 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:57.759886 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:57.760115 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 23:38:57.760486 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":432,"threads_started":1}
I20260810 23:38:57.765609 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:38:57.770098 8164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46343 } }
I20260810 23:38:57.770519 8164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:57.770601 8164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938f578e4ee643d2bf3f3b59c7cf3cc6, State: Initialized, Role: FOLLOWER
I20260810 23:38:57.771136 8164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46343 } }
I20260810 23:38:57.771351 8164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:57.771451 8164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:57.771646 8164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:57.773891 8164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46343 } }
I20260810 23:38:57.774401 8164 leader_election.cc:302] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938f578e4ee643d2bf3f3b59c7cf3cc6; no voters:
I20260810 23:38:57.774983 8164 leader_election.cc:288] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:57.775159 8171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:57.776335 8171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 938f578e4ee643d2bf3f3b59c7cf3cc6, State: Running, Role: LEADER
I20260810 23:38:57.776914 8171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46343 } }
I20260810 23:38:57.777515 8164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:57.779868 8174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 938f578e4ee643d2bf3f3b59c7cf3cc6. Latest consensus state: current_term: 1 leader_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46343 } } }
I20260810 23:38:57.780151 8175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938f578e4ee643d2bf3f3b59c7cf3cc6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46343 } } }
I20260810 23:38:57.780359 8174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:57.780452 8175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:57.782967 8185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:57.788630 8185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:57.798934 8185 catalog_manager.cc:1347] Generated new cluster ID: ba575669cb7d42b8a0a7da1e8af48541
I20260810 23:38:57.799079 8185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:57.815986 8185 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:57.817386 8185 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:57.826845 8185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 938f578e4ee643d2bf3f3b59c7cf3cc6: Generated new TSK 0
I20260810 23:38:57.827883 8185 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:38:57.876765 8214 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 23:38:57.876996 8214 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:38:57.879509 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:57.854571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.30.33.62:46343)
0810 23:38:57.854764 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:57.854771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:57.855319 (+ 548us) server_negotiation.cc:407] Connection header received
0810 23:38:57.855811 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:57.855816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:57.855899 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:57.856044 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:38:57.856670 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:57.857423 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:57.858927 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:57.859390 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:57.859554 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:57.860180 (+ 626us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:38:57.878712 (+ 18532us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:38:57.879154 (+ 442us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 23:38:57.879659 8168 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 23:38:57.881270 8212 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:57.854493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46343 (local address 127.0.0.1:33692)
0810 23:38:57.855179 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:57.855200 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:57.855410 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:57.856106 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:57.856112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:57.856125 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:38:57.856499 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:57.856509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:57.857589 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:57.857594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:57.858762 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:57.858776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:57.859624 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:57.859632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:57.859779 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:57.859852 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:38:57.879076 (+ 19224us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:38:57.879401 (+ 325us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.33.62:46343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":361,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.33.62:46343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34801/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 23:38:57.883200 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8047
I20260810 23:38:57.884565 8157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:57.983035 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8047
2026-08-10T23:38:58Z chronyd exiting
I20260810 23:38:58.039896 30852 test_util.cc:175] -----------------------------------------------
I20260810 23:38:58.040068 30852 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786405091056293-30852-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (796 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 23:38:58.044214 30852 mini_oidc.cc:152] Starting JWKS server
I20260810 23:38:58.046273 30852 webserver.cc:460] Webserver started at https://127.0.0.1:39791/ using document root <none> and password file <none>
I20260810 23:38:58.046530 30852 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:38:58.047266 30852 webserver.cc:460] Webserver started at http://127.0.0.1:35603/ using document root <none> and password file <none>
2026-08-10T23:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:58Z Disabled control of system clock
I20260810 23:38:58.087994 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40353
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:40353
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39791/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/testchainca.cert with env {}
W20260810 23:38:58.453697 8228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:58.454041 8228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:58.454180 8228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:58.461799 8228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:58.461951 8228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:58.462018 8228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:58.462096 8228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:58.462157 8228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:38:58.462247 8228 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:38:58.462298 8228 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39791/jwks.json
I20260810 23:38:58.472769 8228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:40353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39791/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:58.474165 8228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:58.475654 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:58.488343 8238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:58.488474 8240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:58.488433 8242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:58.493434 8228 server_base.cc:1034] running on GCE node
I20260810 23:38:58.494426 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:58.496096 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:58.497376 8228 hybrid_clock.cc:648] HybridClock initialized: now 1786405138497319 us; error 39 us; skew 500 ppm
I20260810 23:38:58.497822 8228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:58.500455 8228 webserver.cc:460] Webserver started at http://127.30.33.62:42343/ using document root <none> and password file <none>
I20260810 23:38:58.501217 8228 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:58.501335 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:58.501688 8228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:58.504838 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "4437abc8a5f64637ac4955641101ef5f"
format_stamp: "Formatted at 2026-08-10 23:38:58 on dist-test-slave-pww0"
server_key: "fc1299025ca451d29d32e333af756ec3"
server_key_iv: "eb827c5063ba16be786cb9981024f8fa"
server_key_version: "encryptionkey@0"
I20260810 23:38:58.505776 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "4437abc8a5f64637ac4955641101ef5f"
format_stamp: "Formatted at 2026-08-10 23:38:58 on dist-test-slave-pww0"
server_key: "fc1299025ca451d29d32e333af756ec3"
server_key_iv: "eb827c5063ba16be786cb9981024f8fa"
server_key_version: "encryptionkey@0"
I20260810 23:38:58.511624 8228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 23:38:58.515357 8254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:58.516876 8228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:58.517047 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "4437abc8a5f64637ac4955641101ef5f"
format_stamp: "Formatted at 2026-08-10 23:38:58 on dist-test-slave-pww0"
server_key: "fc1299025ca451d29d32e333af756ec3"
server_key_iv: "eb827c5063ba16be786cb9981024f8fa"
server_key_version: "encryptionkey@0"
I20260810 23:38:58.517263 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:58.531642 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:58.532569 8228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:58.532892 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:58.550442 8228 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:40353
I20260810 23:38:58.550443 8347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:40353 every 8 connection(s)
I20260810 23:38:58.552065 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:58.556152 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:58.556927 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8228
I20260810 23:38:58.557118 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:58.557783 30852 external_mini_cluster.cc:1428] Setting key d638b328768e7bf8b718c919855f44e9
I20260810 23:38:58.571674 8349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f: Bootstrap starting.
I20260810 23:38:58.573894 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:58.559952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.30.33.62:40353)
0810 23:38:58.561221 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:58.561237 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:58.561304 (+ 67us) server_negotiation.cc:407] Connection header received
0810 23:38:58.563148 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:58.563184 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:58.563942 (+ 758us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:58.564466 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:58.565038 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:58.566926 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:58.568362 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:58.569503 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:58.569686 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:58.570847 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:58.570859 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:58.570879 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:58.571651 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:58.571771 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:58.571802 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:58.571914 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:58.572163 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 23:38:58.572518 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":402,"threads_started":1}
I20260810 23:38:58.575722 8349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:58.577075 8349 log.cc:824] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:58.578437 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:38:58.580322 8349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f: No bootstrap required, opened a new log
I20260810 23:38:58.588771 8349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437abc8a5f64637ac4955641101ef5f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40353 } }
I20260810 23:38:58.589308 8349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:58.589434 8349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4437abc8a5f64637ac4955641101ef5f, State: Initialized, Role: FOLLOWER
I20260810 23:38:58.590071 8349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437abc8a5f64637ac4955641101ef5f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40353 } }
I20260810 23:38:58.590327 8349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:58.590461 8349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:58.590626 8349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:58.592960 8349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437abc8a5f64637ac4955641101ef5f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40353 } }
I20260810 23:38:58.593514 8349 leader_election.cc:302] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4437abc8a5f64637ac4955641101ef5f; no voters:
I20260810 23:38:58.594138 8349 leader_election.cc:288] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:58.594290 8358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:58.595018 8358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [term 1 LEADER]: Becoming Leader. State: Replica: 4437abc8a5f64637ac4955641101ef5f, State: Running, Role: LEADER
I20260810 23:38:58.595638 8358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437abc8a5f64637ac4955641101ef5f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40353 } }
I20260810 23:38:58.596268 8349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:58.598281 8359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4437abc8a5f64637ac4955641101ef5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437abc8a5f64637ac4955641101ef5f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40353 } } }
I20260810 23:38:58.598691 8359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:58.599175 8360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4437abc8a5f64637ac4955641101ef5f. Latest consensus state: current_term: 1 leader_uuid: "4437abc8a5f64637ac4955641101ef5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437abc8a5f64637ac4955641101ef5f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40353 } } }
I20260810 23:38:58.599572 8360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:58.609676 8368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:58.616436 8368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:58.626494 8368 catalog_manager.cc:1347] Generated new cluster ID: bebf23ba3a5742d0a31d1e0b5375fdaa
I20260810 23:38:58.626639 8368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:58.661162 8368 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:58.662530 8368 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:58.669979 8368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4437abc8a5f64637ac4955641101ef5f: Generated new TSK 0
I20260810 23:38:58.670928 8368 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:38:58.706524 8403 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 23:38:58.706661 8403 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:38:58.708086 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:58.686199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.30.33.62:40353)
0810 23:38:58.686436 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:58.686443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:58.686876 (+ 433us) server_negotiation.cc:407] Connection header received
0810 23:38:58.687118 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:58.687125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:58.687231 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:58.687447 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:38:58.688001 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:58.689164 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:58.690821 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:58.691587 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:58.691724 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:58.692307 (+ 583us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:38:58.707270 (+ 14963us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:38:58.707758 (+ 488us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":96}
W20260810 23:38:58.708217 8400 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:58.686035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40353 (local address 127.0.0.1:40314)
0810 23:38:58.686749 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:58.686769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:58.686926 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:58.687446 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:58.687450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:58.687497 (+ 47us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:38:58.687808 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:58.687820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:58.689402 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:58.689409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:58.690644 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:58.690661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:58.691779 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:58.691785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:58.691937 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:58.691978 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:38:58.707618 (+ 15640us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:38:58.707869 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.33.62:40353: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":373,"threads_started":1}
W20260810 23:38:58.708235 8353 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260810 23:38:58.709800 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8228
I20260810 23:38:58.711171 8333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:58.809814 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8228
2026-08-10T23:38:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (829 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 23:38:58.873729 30852 mini_oidc.cc:152] Starting JWKS server
I20260810 23:38:58.875766 30852 webserver.cc:460] Webserver started at https://127.0.0.1:33569/ using document root <none> and password file <none>
I20260810 23:38:58.876041 30852 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:38:58.876772 30852 webserver.cc:460] Webserver started at http://127.0.0.1:38633/ using document root <none> and password file <none>
2026-08-10T23:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:58Z Disabled control of system clock
I20260810 23:38:58.916074 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46601
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:46601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33569/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:38:59.272522 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:59.272862 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:59.272966 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:59.280586 8420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:59.280715 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:59.280753 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:59.280812 8420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:59.280855 8420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:38:59.280916 8420 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:38:59.280956 8420 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33569/jwks.json
I20260810 23:38:59.290516 8420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:46601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33569/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:59.291900 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:59.293733 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:59.306499 8428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:59.306406 8432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:59.306766 8429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:59.307778 8420 server_base.cc:1034] running on GCE node
I20260810 23:38:59.308666 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:59.310184 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:59.311419 8420 hybrid_clock.cc:648] HybridClock initialized: now 1786405139311374 us; error 40 us; skew 500 ppm
I20260810 23:38:59.311829 8420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:38:59.314323 8420 webserver.cc:460] Webserver started at http://127.30.33.62:40551/ using document root <none> and password file <none>
I20260810 23:38:59.315085 8420 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:59.315187 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:59.315563 8420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:59.318650 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "bd56782c82c14f6c827fe352acbea5f4"
format_stamp: "Formatted at 2026-08-10 23:38:59 on dist-test-slave-pww0"
server_key: "f3269b9a3520ac29fc60b0a7f9f578f1"
server_key_iv: "85034d486e8660eee22c07be89377dac"
server_key_version: "encryptionkey@0"
I20260810 23:38:59.319530 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "bd56782c82c14f6c827fe352acbea5f4"
format_stamp: "Formatted at 2026-08-10 23:38:59 on dist-test-slave-pww0"
server_key: "f3269b9a3520ac29fc60b0a7f9f578f1"
server_key_iv: "85034d486e8660eee22c07be89377dac"
server_key_version: "encryptionkey@0"
I20260810 23:38:59.325030 8420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:38:59.328859 8451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:59.330444 8420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:38:59.330650 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "bd56782c82c14f6c827fe352acbea5f4"
format_stamp: "Formatted at 2026-08-10 23:38:59 on dist-test-slave-pww0"
server_key: "f3269b9a3520ac29fc60b0a7f9f578f1"
server_key_iv: "85034d486e8660eee22c07be89377dac"
server_key_version: "encryptionkey@0"
I20260810 23:38:59.330905 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:59.361994 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:59.362951 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:59.363277 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:59.382421 8420 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:46601
I20260810 23:38:59.382448 8538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:46601 every 8 connection(s)
I20260810 23:38:59.383952 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:59.388038 8539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:59.392555 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8420
I20260810 23:38:59.392781 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:38:59.393400 30852 external_mini_cluster.cc:1428] Setting key d90cb1b01f0a8603d64a9a8dd3df52db
I20260810 23:38:59.401147 8539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4: Bootstrap starting.
I20260810 23:38:59.405437 8539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:59.406852 8539 log.cc:824] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:59.410563 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:59.395543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.30.33.62:46601)
0810 23:38:59.396749 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:59.396766 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:59.396827 (+ 61us) server_negotiation.cc:407] Connection header received
0810 23:38:59.399369 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:59.399405 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:59.400104 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:59.400587 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:59.401198 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:59.403412 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:59.404864 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:59.405981 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:59.406198 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:59.407556 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:59.407569 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:59.407593 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:38:59.408204 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:59.408291 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:38:59.408315 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:59.408428 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:59.408681 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 23:38:59.409055 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":349,"threads_started":1}
I20260810 23:38:59.410898 8539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4: No bootstrap required, opened a new log
I20260810 23:38:59.415360 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:38:59.419653 8539 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd56782c82c14f6c827fe352acbea5f4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46601 } }
I20260810 23:38:59.420050 8539 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:38:59.420156 8539 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd56782c82c14f6c827fe352acbea5f4, State: Initialized, Role: FOLLOWER
I20260810 23:38:59.420779 8539 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd56782c82c14f6c827fe352acbea5f4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46601 } }
I20260810 23:38:59.420995 8539 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:38:59.421139 8539 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:38:59.421295 8539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:59.423870 8539 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd56782c82c14f6c827fe352acbea5f4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46601 } }
I20260810 23:38:59.424407 8539 leader_election.cc:302] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd56782c82c14f6c827fe352acbea5f4; no voters:
I20260810 23:38:59.425005 8539 leader_election.cc:288] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:38:59.425148 8546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:59.425904 8546 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [term 1 LEADER]: Becoming Leader. State: Replica: bd56782c82c14f6c827fe352acbea5f4, State: Running, Role: LEADER
I20260810 23:38:59.426507 8546 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd56782c82c14f6c827fe352acbea5f4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46601 } }
I20260810 23:38:59.427124 8539 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:59.430435 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd56782c82c14f6c827fe352acbea5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd56782c82c14f6c827fe352acbea5f4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46601 } } }
I20260810 23:38:59.430935 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:59.430616 8551 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd56782c82c14f6c827fe352acbea5f4. Latest consensus state: current_term: 1 leader_uuid: "bd56782c82c14f6c827fe352acbea5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd56782c82c14f6c827fe352acbea5f4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46601 } } }
I20260810 23:38:59.431686 8551 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:59.432104 8565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:59.437652 8565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:38:59.448493 8565 catalog_manager.cc:1347] Generated new cluster ID: dc0a6932eec94f32b1af7f946f070d16
I20260810 23:38:59.448621 8565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:59.457893 8565 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:59.459324 8565 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:59.466960 8565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd56782c82c14f6c827fe352acbea5f4: Generated new TSK 0
I20260810 23:38:59.467931 8565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:38:59.488863 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:59.480783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.30.33.62:46601)
0810 23:38:59.480985 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:59.480990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:59.481510 (+ 520us) server_negotiation.cc:407] Connection header received
0810 23:38:59.481739 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:59.481744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:59.481825 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:59.481968 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:38:59.482488 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:59.483294 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:59.484712 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:59.485163 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:59.485289 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:59.485838 (+ 549us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:38:59.488204 (+ 2366us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33569/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:38:59.488564 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33569/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 23:38:59.489004 8544 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33569/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 23:38:59.491072 8589 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:59.480663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46601 (local address 127.0.0.1:60066)
0810 23:38:59.481385 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:59.481401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:59.481556 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:59.482005 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:59.482010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:59.482022 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:38:59.482320 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:59.482331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:59.483451 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:59.483479 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:59.484558 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:59.484572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:59.485371 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:59.485378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:59.485521 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:59.485559 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:38:59.488459 (+ 2900us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33569/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:38:59.490620 (+ 2161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.33.62:46601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33569/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":1814,"thread_start_us":456,"threads_started":1}
I20260810 23:38:59.493177 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8420
I20260810 23:38:59.494623 8529 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 23:38:59.503496 8593 webserver.cc:533] Webserver: sslize failed:
E20260810 23:38:59.503646 8593 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:38:59.591053 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8420
2026-08-10T23:38:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (781 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T23:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:59Z Disabled control of system clock
I20260810 23:38:59.693543 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:38091
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:38091
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/testchainca.cert with env {}
W20260810 23:39:00.046247 8604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:00.046582 8604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:00.046675 8604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:00.054343 8604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:00.054474 8604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:00.054526 8604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:00.054584 8604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:00.054628 8604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:39:00.054690 8604 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:39:00.054730 8604 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 23:39:00.065308 8604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:38091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:00.067203 8604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:00.068826 8604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:00.081918 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:00.081887 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:00.081954 8613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:00.082782 8604 server_base.cc:1034] running on GCE node
I20260810 23:39:00.083729 8604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:00.085361 8604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:00.086582 8604 hybrid_clock.cc:648] HybridClock initialized: now 1786405140086534 us; error 44 us; skew 500 ppm
I20260810 23:39:00.086966 8604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:00.089480 8604 webserver.cc:460] Webserver started at http://127.30.33.62:38589/ using document root <none> and password file <none>
I20260810 23:39:00.090273 8604 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:00.090394 8604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:00.090714 8604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:00.093952 8604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "5fb555ad3d72492093f95675b315dd92"
format_stamp: "Formatted at 2026-08-10 23:39:00 on dist-test-slave-pww0"
server_key: "cc8a03a8f56db6cefeaa95adcd8951a4"
server_key_iv: "ed77f89e8c93e961a0132dca3028f23a"
server_key_version: "encryptionkey@0"
I20260810 23:39:00.094832 8604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "5fb555ad3d72492093f95675b315dd92"
format_stamp: "Formatted at 2026-08-10 23:39:00 on dist-test-slave-pww0"
server_key: "cc8a03a8f56db6cefeaa95adcd8951a4"
server_key_iv: "ed77f89e8c93e961a0132dca3028f23a"
server_key_version: "encryptionkey@0"
I20260810 23:39:00.100749 8604 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 23:39:00.104625 8627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:00.106165 8604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:39:00.106370 8604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "5fb555ad3d72492093f95675b315dd92"
format_stamp: "Formatted at 2026-08-10 23:39:00 on dist-test-slave-pww0"
server_key: "cc8a03a8f56db6cefeaa95adcd8951a4"
server_key_iv: "ed77f89e8c93e961a0132dca3028f23a"
server_key_version: "encryptionkey@0"
I20260810 23:39:00.106604 8604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:00.121846 8604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:00.122779 8604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:00.123067 8604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:00.141911 8604 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:38091
I20260810 23:39:00.141932 8720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:38091 every 8 connection(s)
I20260810 23:39:00.143419 8604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:00.147908 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:00.151278 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8604
I20260810 23:39:00.151558 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:00.152506 30852 external_mini_cluster.cc:1428] Setting key e6a02982df479ce4d480bf87e7a37b8e
I20260810 23:39:00.162020 8721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92: Bootstrap starting.
I20260810 23:39:00.165861 8721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:00.167032 8721 log.cc:824] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:00.167445 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:00.155014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.30.33.62:38091)
0810 23:39:00.156161 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:00.156174 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:00.156222 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:39:00.158062 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:00.158088 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:00.158642 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:00.159087 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:00.159808 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:00.161122 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:00.162542 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:00.163309 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:00.163550 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:00.164829 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:00.164838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:00.164855 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:00.165447 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:00.165551 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:00.165576 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:00.165685 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:00.165964 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:39:00.166370 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":414,"threads_started":1}
I20260810 23:39:00.170596 8721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92: No bootstrap required, opened a new log
I20260810 23:39:00.171922 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:39:00.179962 8721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb555ad3d72492093f95675b315dd92" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38091 } }
I20260810 23:39:00.180377 8721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:00.180459 8721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb555ad3d72492093f95675b315dd92, State: Initialized, Role: FOLLOWER
I20260810 23:39:00.181048 8721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb555ad3d72492093f95675b315dd92" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38091 } }
I20260810 23:39:00.181282 8721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:00.181391 8721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:00.181526 8721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:00.184042 8721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb555ad3d72492093f95675b315dd92" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38091 } }
I20260810 23:39:00.184563 8721 leader_election.cc:302] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fb555ad3d72492093f95675b315dd92; no voters:
I20260810 23:39:00.185185 8721 leader_election.cc:288] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:00.185353 8729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:00.186098 8729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [term 1 LEADER]: Becoming Leader. State: Replica: 5fb555ad3d72492093f95675b315dd92, State: Running, Role: LEADER
I20260810 23:39:00.186637 8729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb555ad3d72492093f95675b315dd92" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38091 } }
I20260810 23:39:00.187337 8721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:00.189823 8731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fb555ad3d72492093f95675b315dd92. Latest consensus state: current_term: 1 leader_uuid: "5fb555ad3d72492093f95675b315dd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb555ad3d72492093f95675b315dd92" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38091 } } }
I20260810 23:39:00.189795 8730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fb555ad3d72492093f95675b315dd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb555ad3d72492093f95675b315dd92" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 38091 } } }
I20260810 23:39:00.190285 8730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:00.190285 8731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:00.191219 8743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:00.197667 8743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:00.208321 8743 catalog_manager.cc:1347] Generated new cluster ID: 449eb27656d54f928d70c1e2e9d4c6eb
I20260810 23:39:00.208446 8743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:00.219960 8743 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:00.221441 8743 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:00.230012 8743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fb555ad3d72492093f95675b315dd92: Generated new TSK 0
I20260810 23:39:00.230958 8743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:39:00.691740 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:00.254892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.30.33.62:38091)
0810 23:39:00.255127 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:00.255135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:00.255707 (+ 572us) server_negotiation.cc:407] Connection header received
0810 23:39:00.255978 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:00.255985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:00.256103 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:00.256302 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:39:00.256835 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:00.257991 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:00.259544 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:00.260259 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:00.260407 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:00.260933 (+ 526us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:39:00.691087 (+430154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:39:00.691444 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 23:39:00.691849 8726 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 23:39:00.692183 8765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:00.254860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:38091 (local address 127.0.0.1:52658)
0810 23:39:00.255565 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:00.255587 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:00.255769 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:00.256351 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:00.256356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:00.256369 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:39:00.256662 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:00.256672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:00.258210 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:00.258216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:00.259354 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:00.259368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:00.260441 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:00.260446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:00.260595 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:00.260636 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:39:00.691404 (+430768us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:39:00.691719 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.33.62:38091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":348,"threads_started":1}
I20260810 23:39:00.693547 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8604
I20260810 23:39:00.694697 8714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:00.802186 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8604
2026-08-10T23:39:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1192 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (449 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1527 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:39:02 dist-test-slave-pww0 krb5kdc[8811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:39:02 dist-test-slave-pww0 krb5kdc[8811](info): set up 2 sockets
Aug 10 23:39:02 dist-test-slave-pww0 krb5kdc[8811](info): commencing operation
krb5kdc: starting...
W20260810 23:39:04.082746 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:39:04 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:39:04.361055 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:42695
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:42695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:04.712013 8832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:04.712345 8832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:04.712492 8832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:04.719949 8832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:04.720079 8832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:04.720156 8832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:04.720232 8832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:04.720279 8832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:04.730846 8832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:42695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:42695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:04.732290 8832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:04.733659 8832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:04.746364 8841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:04.746430 8842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:04.746450 8845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:04.747000 8832 server_base.cc:1034] running on GCE node
I20260810 23:39:04.747979 8832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:04.749598 8832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:04.750878 8832 hybrid_clock.cc:648] HybridClock initialized: now 1786405144750818 us; error 48 us; skew 500 ppm
I20260810 23:39:04.751437 8832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:04 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405144, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:04.754926 8832 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:39:04.756705 8832 webserver.cc:460] Webserver started at http://127.30.33.62:35973/ using document root <none> and password file <none>
I20260810 23:39:04.757498 8832 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:04.757664 8832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:04.757987 8832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:04.761384 8832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "fb2d2964f49e4c468976ebb19b1fb9c6"
format_stamp: "Formatted at 2026-08-10 23:39:04 on dist-test-slave-pww0"
server_key: "2491fe29763be6fbcdbf00cfe7906ae2"
server_key_iv: "c4038fd7356f50c4c730c95fd332b708"
server_key_version: "encryptionkey@0"
I20260810 23:39:04.762212 8832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "fb2d2964f49e4c468976ebb19b1fb9c6"
format_stamp: "Formatted at 2026-08-10 23:39:04 on dist-test-slave-pww0"
server_key: "2491fe29763be6fbcdbf00cfe7906ae2"
server_key_iv: "c4038fd7356f50c4c730c95fd332b708"
server_key_version: "encryptionkey@0"
I20260810 23:39:04.768136 8832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 23:39:04.772171 8856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:04.773689 8832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:39:04.773895 8832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "fb2d2964f49e4c468976ebb19b1fb9c6"
format_stamp: "Formatted at 2026-08-10 23:39:04 on dist-test-slave-pww0"
server_key: "2491fe29763be6fbcdbf00cfe7906ae2"
server_key_iv: "c4038fd7356f50c4c730c95fd332b708"
server_key_version: "encryptionkey@0"
I20260810 23:39:04.774103 8832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:04.790880 8832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:04.795637 8832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:04.795946 8832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:04.814985 8832 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:42695
I20260810 23:39:04.815011 8955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:42695 every 8 connection(s)
I20260810 23:39:04.816475 8832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:04.819145 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8832
I20260810 23:39:04.819371 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:04.820062 30852 external_mini_cluster.cc:1428] Setting key 0ebbd4035c11ccd1e7952ae5cdba40c8
I20260810 23:39:04.820595 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:39:04 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:04.835852 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6: Bootstrap starting.
I20260810 23:39:04.839735 8956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:04.841086 8956 log.cc:824] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:04.844583 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6: No bootstrap required, opened a new log
I20260810 23:39:04.845016 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:04.822074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.30.33.62:42695)
0810 23:39:04.822887 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:04.822905 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:04.822960 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:39:04.824712 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:04.824745 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:04.825333 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:04.825831 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:04.827381 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:04.829490 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:04.830969 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:04.832127 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:04.832340 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:04.836872 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:04.836882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:04.836900 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:04.836951 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:04.840234 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:04.841249 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:04.841259 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:04.841269 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:04.841475 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:04.841943 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:04.841948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:04.841952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:04.842710 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:04.842952 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:04.843443 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 23:39:04.843895 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":319,"threads_started":1}
I20260810 23:39:04.852696 8956 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 42695 } }
I20260810 23:39:04.853121 8956 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:04.853231 8956 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2d2964f49e4c468976ebb19b1fb9c6, State: Initialized, Role: FOLLOWER
I20260810 23:39:04.853861 8956 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 42695 } }
I20260810 23:39:04.854115 8956 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:04.854279 8956 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:04.854496 8956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:04.857005 8956 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 42695 } }
I20260810 23:39:04.857630 8956 leader_election.cc:302] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2d2964f49e4c468976ebb19b1fb9c6; no voters:
I20260810 23:39:04.858273 8956 leader_election.cc:288] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:04.858479 8963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:04.859576 8963 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [term 1 LEADER]: Becoming Leader. State: Replica: fb2d2964f49e4c468976ebb19b1fb9c6, State: Running, Role: LEADER
I20260810 23:39:04.860157 8963 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 42695 } }
I20260810 23:39:04.860625 8956 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:04.863190 8967 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2d2964f49e4c468976ebb19b1fb9c6. Latest consensus state: current_term: 1 leader_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 42695 } } }
I20260810 23:39:04.863394 8965 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2d2964f49e4c468976ebb19b1fb9c6" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 42695 } } }
I20260810 23:39:04.863853 8965 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:04.863853 8967 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:04.865923 8976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:04.872534 8976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:04.884833 8976 catalog_manager.cc:1347] Generated new cluster ID: 1a0ad89631f54cdb83c3ca67cf704d3e
I20260810 23:39:04.884982 8976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:04.903834 8976 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:04.905910 8976 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:04.919435 8976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb2d2964f49e4c468976ebb19b1fb9c6: Generated new TSK 0
I20260810 23:39:04.920665 8976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:39:04.991317 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:42695
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:05.355588 9000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:05.355944 9000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:05.356091 9000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:05.363971 9000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:05.364123 9000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:05.364346 9000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:05.375105 9000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:42695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:05.376554 9000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:05.377995 9000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:05.392576 9007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:05.392611 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:05.392525 9015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:05.393553 9000 server_base.cc:1034] running on GCE node
I20260810 23:39:05.394248 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:05.395346 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:05.396606 9000 hybrid_clock.cc:648] HybridClock initialized: now 1786405145396527 us; error 58 us; skew 500 ppm
I20260810 23:39:05.396983 9000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:05 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:05.400609 9000 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:39:05.402390 9000 webserver.cc:460] Webserver started at http://127.30.33.1:46353/ using document root <none> and password file <none>
I20260810 23:39:05.403148 9000 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:05.403280 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:05.403664 9000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:05.406798 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "6cfbc637d89d4560a382534891f5a3f5"
format_stamp: "Formatted at 2026-08-10 23:39:05 on dist-test-slave-pww0"
server_key: "591f7c316192be284efe59477aa55a44"
server_key_iv: "9e5d2bac549775a1f37e9b166876a7bc"
server_key_version: "encryptionkey@0"
I20260810 23:39:05.407676 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "6cfbc637d89d4560a382534891f5a3f5"
format_stamp: "Formatted at 2026-08-10 23:39:05 on dist-test-slave-pww0"
server_key: "591f7c316192be284efe59477aa55a44"
server_key_iv: "9e5d2bac549775a1f37e9b166876a7bc"
server_key_version: "encryptionkey@0"
I20260810 23:39:05.413936 9000 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:39:05.417954 9025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:05.419848 9000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:39:05.420035 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "6cfbc637d89d4560a382534891f5a3f5"
format_stamp: "Formatted at 2026-08-10 23:39:05 on dist-test-slave-pww0"
server_key: "591f7c316192be284efe59477aa55a44"
server_key_iv: "9e5d2bac549775a1f37e9b166876a7bc"
server_key_version: "encryptionkey@0"
I20260810 23:39:05.420258 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:05.443322 9000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:05.448311 9000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:05.448617 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:05.449679 9000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:05.451227 9000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:05.451315 9000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:05.451442 9000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:05.451514 9000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 23:39:05.484443 9000 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:35685
I20260810 23:39:05.484532 9235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:35685 every 8 connection(s)
I20260810 23:39:05.485816 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:05.490899 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 9000
I20260810 23:39:05.491106 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:39:05.491804 30852 external_mini_cluster.cc:1428] Setting key 7335561b4bb8940264d4736d508f706e
Aug 10 23:39:05 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:05.510329 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:05.488581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:36551 (local address 127.30.33.62:42695)
0810 23:39:05.489098 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:05.489105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:05.490265 (+ 1160us) server_negotiation.cc:407] Connection header received
0810 23:39:05.492334 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:05.492340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:05.492433 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:05.492560 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:05.494549 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:05.495307 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:05.497812 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:05.498261 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:05.498382 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:05.503141 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:05.503148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:05.503152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:05.503195 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:05.506462 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:05.507500 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:05.507508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:05.507512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:05.507720 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:05.508269 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:05.508275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:05.508279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:05.508654 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:05.508827 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:05.509677 (+ 850us) server_negotiation.cc:299] Negotiation successful
0810 23:39:05.509950 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":238,"threads_started":1}
I20260810 23:39:05.511222 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:05.489193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:42695 (local address 127.30.33.1:36551)
0810 23:39:05.490050 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:05.490101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:05.492002 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:05.492872 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:05.492882 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:05.493486 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:05.494316 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:05.494351 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:05.495526 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:05.495532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:05.497606 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:05.497626 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:05.498535 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:05.498542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:05.498867 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:05.499749 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:05.499789 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:05.502791 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:05.506694 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:05.506704 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:05.506721 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:05.507268 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:05.507881 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:05.507885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:05.507888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:05.508133 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:05.508827 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:05.508834 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:05.509232 (+ 398us) client_negotiation.cc:769] Sending connection context
0810 23:39:05.509500 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 23:39:05.509978 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":346,"threads_started":1}
I20260810 23:39:05.513144 9236 heartbeater.cc:347] Connected to a master server at 127.30.33.62:42695
I20260810 23:39:05.513537 9236 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:05.514544 9236 heartbeater.cc:511] Master 127.30.33.62:42695 requested a full tablet report, sending...
I20260810 23:39:05.517401 8888 ts_manager.cc:194] Registered new tserver with Master: 6cfbc637d89d4560a382534891f5a3f5 (127.30.33.1:35685)
I20260810 23:39:05.519003 8888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:36551
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:39:05.558792 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:42695
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:05.915402 9250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:05.915776 9250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:05.915932 9250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:05.923834 9250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:05.923990 9250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:05.924150 9250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:05.935037 9250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:42695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:05.936544 9250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:05.937971 9250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:05.952355 9262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:05.952484 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:05.952512 9261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:05.953784 9250 server_base.cc:1034] running on GCE node
I20260810 23:39:05.954641 9250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:05.955824 9250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:05.957094 9250 hybrid_clock.cc:648] HybridClock initialized: now 1786405145957006 us; error 77 us; skew 500 ppm
I20260810 23:39:05.957530 9250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:05 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:05.961258 9250 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:39:05.963050 9250 webserver.cc:460] Webserver started at http://127.30.33.2:45903/ using document root <none> and password file <none>
I20260810 23:39:05.963893 9250 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:05.964017 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:05.964344 9250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:05.967785 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "7526153f95ec493d8442f623b919d255"
format_stamp: "Formatted at 2026-08-10 23:39:05 on dist-test-slave-pww0"
server_key: "0190a43db374fad5e9689282b3c80368"
server_key_iv: "3e3eba949c068f3a744124f17395e706"
server_key_version: "encryptionkey@0"
I20260810 23:39:05.968662 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "7526153f95ec493d8442f623b919d255"
format_stamp: "Formatted at 2026-08-10 23:39:05 on dist-test-slave-pww0"
server_key: "0190a43db374fad5e9689282b3c80368"
server_key_iv: "3e3eba949c068f3a744124f17395e706"
server_key_version: "encryptionkey@0"
I20260810 23:39:05.974634 9250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:39:05.978818 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:05.980486 9250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:05.980674 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "7526153f95ec493d8442f623b919d255"
format_stamp: "Formatted at 2026-08-10 23:39:05 on dist-test-slave-pww0"
server_key: "0190a43db374fad5e9689282b3c80368"
server_key_iv: "3e3eba949c068f3a744124f17395e706"
server_key_version: "encryptionkey@0"
I20260810 23:39:05.980914 9250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:05.999882 9250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:06.004606 9250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:06.004925 9250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:06.006055 9250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:06.007682 9250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:06.007762 9250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:06.007889 9250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:06.007941 9250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:06.040871 9250 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:33071
I20260810 23:39:06.040904 9495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:33071 every 8 connection(s)
I20260810 23:39:06.042304 9250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:06.048189 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 9250
I20260810 23:39:06.048396 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:39:06.048942 30852 external_mini_cluster.cc:1428] Setting key 2bba8e17995ed0ffc342b8a899e22942
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:06.066748 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.045045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:41823 (local address 127.30.33.62:42695)
0810 23:39:06.045510 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.045514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.046508 (+ 994us) server_negotiation.cc:407] Connection header received
0810 23:39:06.048770 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.048776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.048858 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.048961 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.051015 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.051881 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.054616 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.055090 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.055212 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.059932 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:06.059939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:06.059943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:06.059987 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:06.062885 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.064032 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.064036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.064039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.064189 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.064721 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.064724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.064727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.065018 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:06.065138 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:06.066133 (+ 995us) server_negotiation.cc:299] Negotiation successful
0810 23:39:06.066416 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":252,"threads_started":1}
I20260810 23:39:06.068267 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.045548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:42695 (local address 127.30.33.2:41823)
0810 23:39:06.046324 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:06.046373 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:06.048411 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:06.049285 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:06.049299 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:06.050079 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:06.050836 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:06.050866 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.052055 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.052061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:06.054377 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:06.054407 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.055369 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.055376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:06.055721 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.056540 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:06.056582 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:06.059598 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:06.063100 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:06.063112 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:06.063142 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:06.063904 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:06.064311 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:06.064315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:06.064319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:06.064620 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:06.065150 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:06.065159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:06.065687 (+ 528us) client_negotiation.cc:769] Sending connection context
0810 23:39:06.066013 (+ 326us) client_negotiation.cc:240] Negotiation successful
0810 23:39:06.066668 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":307,"threads_started":1}
I20260810 23:39:06.069967 9496 heartbeater.cc:347] Connected to a master server at 127.30.33.62:42695
I20260810 23:39:06.070335 9496 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:06.071178 9496 heartbeater.cc:511] Master 127.30.33.62:42695 requested a full tablet report, sending...
I20260810 23:39:06.072930 8888 ts_manager.cc:194] Registered new tserver with Master: 7526153f95ec493d8442f623b919d255 (127.30.33.2:33071)
I20260810 23:39:06.073798 8888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:41823
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:39:06.114744 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:42695
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:06.466985 9506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:06.467293 9506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:06.467402 9506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:06.474988 9506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:06.475126 9506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:06.475277 9506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:06.486835 9506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:42695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:06.488632 9506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:06.490451 9506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:06.504213 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:06.504196 9517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:06.504498 9516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:06.505179 9506 server_base.cc:1034] running on GCE node
I20260810 23:39:06.505862 9506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:06.506910 9506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:06.508138 9506 hybrid_clock.cc:648] HybridClock initialized: now 1786405146508063 us; error 57 us; skew 500 ppm
I20260810 23:39:06.508500 9506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.511909 9506 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:39:06.513572 9506 webserver.cc:460] Webserver started at http://127.30.33.3:35445/ using document root <none> and password file <none>
I20260810 23:39:06.514333 9506 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:06.514468 9506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:06.514804 9506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:06.517975 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "cb9888124fc04156b432ef56aee2a6d9"
format_stamp: "Formatted at 2026-08-10 23:39:06 on dist-test-slave-pww0"
server_key: "80a72e31e749be5138b7d93998396758"
server_key_iv: "5b6859ed37a77d8d48ab9779754c7b85"
server_key_version: "encryptionkey@0"
I20260810 23:39:06.518828 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "cb9888124fc04156b432ef56aee2a6d9"
format_stamp: "Formatted at 2026-08-10 23:39:06 on dist-test-slave-pww0"
server_key: "80a72e31e749be5138b7d93998396758"
server_key_iv: "5b6859ed37a77d8d48ab9779754c7b85"
server_key_version: "encryptionkey@0"
I20260810 23:39:06.523144 9236 heartbeater.cc:503] Master 127.30.33.62:42695 was elected leader, sending a full tablet report...
I20260810 23:39:06.524963 9506 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 23:39:06.529066 9534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:06.530620 9506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:39:06.530794 9506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "cb9888124fc04156b432ef56aee2a6d9"
format_stamp: "Formatted at 2026-08-10 23:39:06 on dist-test-slave-pww0"
server_key: "80a72e31e749be5138b7d93998396758"
server_key_iv: "5b6859ed37a77d8d48ab9779754c7b85"
server_key_version: "encryptionkey@0"
I20260810 23:39:06.531019 9506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:06.553817 9506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:06.558651 9506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:06.558966 9506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:06.560111 9506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:06.561678 9506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:06.561762 9506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:06.561883 9506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:06.561933 9506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:06.593660 9506 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:45599
I20260810 23:39:06.593725 9742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:45599 every 8 connection(s)
I20260810 23:39:06.595044 9506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:06.603609 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 9506
I20260810 23:39:06.603785 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:39:06.604264 30852 external_mini_cluster.cc:1428] Setting key aa8d041bcd63947b129df313b2134d72
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:06.616168 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.597964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:60017 (local address 127.30.33.62:42695)
0810 23:39:06.598459 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.598465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.599543 (+ 1078us) server_negotiation.cc:407] Connection header received
0810 23:39:06.601503 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.601510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.601598 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.601698 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.603620 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.604559 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.606501 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.606954 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.607054 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.610719 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:06.610724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:06.610728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:06.610768 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:06.613269 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.614081 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.614085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.614088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.614232 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.614592 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.614595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.614597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.614869 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:06.614998 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:06.615690 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 23:39:06.615875 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260810 23:39:06.617204 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.598493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:42695 (local address 127.30.33.3:60017)
0810 23:39:06.599326 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:06.599378 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:06.601171 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:06.601989 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:06.601998 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:06.602588 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:06.603359 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:06.603397 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.604751 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.604757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:06.606364 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:06.606378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.607200 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.607208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:06.607438 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.608167 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:06.608194 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:06.610467 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:06.613451 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:06.613460 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:06.613475 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:06.613968 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:06.614315 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:06.614318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:06.614320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:06.614511 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:06.614963 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:06.614969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:06.615322 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 23:39:06.615604 (+ 282us) client_negotiation.cc:240] Negotiation successful
0810 23:39:06.616052 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":333,"threads_started":1}
I20260810 23:39:06.618716 9744 heartbeater.cc:347] Connected to a master server at 127.30.33.62:42695
I20260810 23:39:06.619050 9744 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:06.619865 9744 heartbeater.cc:511] Master 127.30.33.62:42695 requested a full tablet report, sending...
I20260810 23:39:06.621474 8888 ts_manager.cc:194] Registered new tserver with Master: cb9888124fc04156b432ef56aee2a6d9 (127.30.33.3:45599)
I20260810 23:39:06.622304 8888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:60017
I20260810 23:39:06.632745 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/test-admin.keytab.
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.666898 30852 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 23:39:06.672425 9758 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:06.669858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.30.33.1:35685)
0810 23:39:06.670494 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.670501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.670553 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:39:06.670619 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.670625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.670839 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.670966 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.671992 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44682: BlockingRecv error: recv got EOF from 127.0.0.1:44682 (error 108)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":311,"threads_started":1}
W20260810 23:39:06.673409 30852 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc)
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.675050 30852 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.676440 30852 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:39:06.693415 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.678474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.30.33.1:35685)
0810 23:39:06.678698 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.678705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.678957 (+ 252us) server_negotiation.cc:407] Connection header received
0810 23:39:06.679118 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.679123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.679197 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.679289 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.680490 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.681507 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.682802 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.683423 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.683611 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.687572 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:06.687590 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:06.687606 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:06.687647 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:06.690570 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.691327 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.691336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.691338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.691552 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.691875 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.691878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.691880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.692270 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:06.692490 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:06.692911 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 23:39:06.693133 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 23:39:06.699900 9758 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:06.697978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.30.33.1:35685)
0810 23:39:06.698196 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.698202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.698536 (+ 334us) server_negotiation.cc:407] Connection header received
0810 23:39:06.698706 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.698710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.698778 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.698856 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.699665 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44692: BlockingRecv error: recv got EOF from 127.0.0.1:44692 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 23:39:06.700035 30852 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc)
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.701532 30852 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.702921 30852 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:39:06.717818 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.704418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.30.33.1:35685)
0810 23:39:06.704632 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.704638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.704842 (+ 204us) server_negotiation.cc:407] Connection header received
0810 23:39:06.704988 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.704992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.705057 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.705142 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.706156 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.706895 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.708212 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.708669 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.708767 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.712589 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:06.712595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:06.712599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:06.712641 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:06.715269 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.716053 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.716058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.716061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.716223 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.716564 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.716572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.716574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.716863 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:06.717063 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:06.717303 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 23:39:06.717494 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 23:39:06.722440 9758 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:06.720716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.30.33.1:35685)
0810 23:39:06.720920 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.720927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.721151 (+ 224us) server_negotiation.cc:407] Connection header received
0810 23:39:06.721283 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.721287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.721376 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.721459 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.722200 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44702: BlockingRecv error: recv got EOF from 127.0.0.1:44702 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 23:39:06.722556 30852 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786405091056293-30852-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.724210 30852 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:06.725585 30852 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:39:06.740458 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.727078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.30.33.1:35685)
0810 23:39:06.727298 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.727304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.727572 (+ 268us) server_negotiation.cc:407] Connection header received
0810 23:39:06.727708 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.727713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.727780 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.727862 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.728772 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.729513 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.730772 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.731211 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.731309 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.734902 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:06.734909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:06.734913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:06.734961 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:06.737593 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.738316 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.738320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.738323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.738486 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.738926 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.738931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.738933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.739270 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:06.739587 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:06.739828 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:39:06.740087 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:39:06.742224 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 9000
I20260810 23:39:06.755992 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.742939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.30.33.1:35685)
0810 23:39:06.743115 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.743122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.743484 (+ 362us) server_negotiation.cc:407] Connection header received
0810 23:39:06.743642 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.743646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.743715 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.743817 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.744915 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.745631 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.747092 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.747614 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.747735 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.749942 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:06.749947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:06.749950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:06.749988 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:06.752771 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.753626 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.753633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.753637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.753823 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.754264 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.754271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.754275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.754611 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:06.754905 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:06.755303 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 23:39:06.755626 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:39:06.756738 9230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:06.843130 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 9000
I20260810 23:39:06.864487 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 9250
Aug 10 23:39:06 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:39:06.881845 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.865504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.30.33.2:33071)
0810 23:39:06.866162 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:06.866169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:06.866198 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:39:06.866303 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:06.866313 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:06.866542 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:06.866688 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:06.867850 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.868916 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.870239 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:06.870882 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:06.870996 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:06.875175 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:06.875197 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:06.875214 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:06.875261 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:06.878547 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.879435 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.879445 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.879447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.879666 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:06.880101 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:06.880106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:06.880108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:06.880538 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:06.880822 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:06.881251 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 23:39:06.881517 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":344,"threads_started":1}
I20260810 23:39:06.883279 9488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:06.971361 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 9250
I20260810 23:39:06.998682 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 9506
Aug 10 23:39:07 dist-test-slave-pww0 krb5kdc[8811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:39:07.016907 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:06.999786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.30.33.3:45599)
0810 23:39:07.000445 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:07.000453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:07.000483 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:39:07.000569 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:07.000575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:07.000803 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:07.000960 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:07.002129 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:07.003194 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:07.004738 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:07.005368 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:07.005503 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:07.009713 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:07.009729 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:07.009745 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:07.009791 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:07.013101 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:07.014168 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:07.014180 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:07.014182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:07.014381 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:07.014942 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:07.014946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:07.014949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:07.015419 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:07.015836 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:07.016227 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 23:39:07.016568 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":331,"threads_started":1}
I20260810 23:39:07.018427 9735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:07.104591 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 9506
I20260810 23:39:07.125214 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8832
I20260810 23:39:07.126708 8950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:07.236644 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8832
2026-08-10T23:39:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4459 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:39:07 dist-test-slave-pww0 krb5kdc[9794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:39:07 dist-test-slave-pww0 krb5kdc[9794](info): set up 2 sockets
Aug 10 23:39:07 dist-test-slave-pww0 krb5kdc[9794](info): commencing operation
krb5kdc: starting...
W20260810 23:39:08.554859 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:39:08 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab.
Entry for principal oryx/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab.
I20260810 23:39:08.836277 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40579
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab
--principal=oryx/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:09.193156 9814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:09.193485 9814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:09.193624 9814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:09.201200 9814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:09.201344 9814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:09.201421 9814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:09.201493 9814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:09.201545 9814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:09.211983 9814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab
--principal=oryx/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:09.213443 9814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:09.214912 9814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:09.227547 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:09.227427 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:09.227742 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:09.228598 9814 server_base.cc:1034] running on GCE node
I20260810 23:39:09.229473 9814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:09.231096 9814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:09.232347 9814 hybrid_clock.cc:648] HybridClock initialized: now 1786405149232296 us; error 42 us; skew 500 ppm
I20260810 23:39:09.232765 9814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:09 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405149, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:09.236171 9814 init.cc:377] Logged in from keytab as oryx/127.30.33.62@KRBTEST.COM (short username oryx)
I20260810 23:39:09.237906 9814 webserver.cc:460] Webserver started at http://127.30.33.62:46049/ using document root <none> and password file <none>
I20260810 23:39:09.238639 9814 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:09.238760 9814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:09.239073 9814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:09.242318 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "97a992b3716d4a66b6ca82dd811968fd"
format_stamp: "Formatted at 2026-08-10 23:39:09 on dist-test-slave-pww0"
server_key: "94c96a3f95ec1c6d198f23cc7614f9ff"
server_key_iv: "8be736724d8afbbbdd23488d0fdfc8ba"
server_key_version: "encryptionkey@0"
I20260810 23:39:09.243145 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "97a992b3716d4a66b6ca82dd811968fd"
format_stamp: "Formatted at 2026-08-10 23:39:09 on dist-test-slave-pww0"
server_key: "94c96a3f95ec1c6d198f23cc7614f9ff"
server_key_iv: "8be736724d8afbbbdd23488d0fdfc8ba"
server_key_version: "encryptionkey@0"
I20260810 23:39:09.249083 9814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:39:09.253726 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:09.255491 9814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:39:09.255745 9814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "97a992b3716d4a66b6ca82dd811968fd"
format_stamp: "Formatted at 2026-08-10 23:39:09 on dist-test-slave-pww0"
server_key: "94c96a3f95ec1c6d198f23cc7614f9ff"
server_key_iv: "8be736724d8afbbbdd23488d0fdfc8ba"
server_key_version: "encryptionkey@0"
I20260810 23:39:09.255980 9814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:09.286312 9814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:09.290650 9814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:09.290962 9814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:09.309930 9814 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:40579
I20260810 23:39:09.309967 9928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:40579 every 8 connection(s)
I20260810 23:39:09.311519 9814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:09.314911 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 9814
I20260810 23:39:09.315110 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:09.315740 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:09.315842 30852 external_mini_cluster.cc:1428] Setting key bee34015bfc6364733a509e65c3ed3d5
I20260810 23:39:09.331542 9930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: Bootstrap starting.
Aug 10 23:39:09 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:09.335425 9930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:09.336752 9930 log.cc:824] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:09.340499 9930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: No bootstrap required, opened a new log
I20260810 23:39:09.341248 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:09.318005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.30.33.62:40579)
0810 23:39:09.318898 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:09.318911 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:09.318957 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:39:09.320724 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:09.320752 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:09.321202 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:09.321585 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:09.323722 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:09.325169 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:09.327014 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:09.327857 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:09.328016 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:09.333018 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:09.333028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:09.333044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:09.333090 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:09.336344 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:09.337334 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:09.337345 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:09.337355 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:09.337627 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:09.338125 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:09.338130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:09.338133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:09.338761 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:09.339000 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:09.339615 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 23:39:09.340029 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":386,"threads_started":1}
I20260810 23:39:09.348894 9930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } }
I20260810 23:39:09.349406 9930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:09.349519 9930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a992b3716d4a66b6ca82dd811968fd, State: Initialized, Role: FOLLOWER
I20260810 23:39:09.350185 9930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } }
I20260810 23:39:09.350456 9930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:09.350592 9930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:09.350749 9930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:09.353482 9930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } }
I20260810 23:39:09.354080 9930 leader_election.cc:302] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97a992b3716d4a66b6ca82dd811968fd; no voters:
I20260810 23:39:09.354732 9930 leader_election.cc:288] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:09.354871 9941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:09.355576 9941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 1 LEADER]: Becoming Leader. State: Replica: 97a992b3716d4a66b6ca82dd811968fd, State: Running, Role: LEADER
I20260810 23:39:09.356187 9941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } }
I20260810 23:39:09.356879 9930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:09.359074 9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97a992b3716d4a66b6ca82dd811968fd. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } }
I20260810 23:39:09.359426 9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:09.362382 9956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:09.362414 9942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } }
I20260810 23:39:09.362876 9942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:09.367352 9956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:09.376920 9956 catalog_manager.cc:1347] Generated new cluster ID: 97c24683486b4c94963bd5fc148581c3
I20260810 23:39:09.377066 9956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:09.394665 9956 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:09.396397 9956 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:09.402217 9956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: Generated new TSK 0
I20260810 23:39:09.403144 9956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab.
Entry for principal oryx/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab.
I20260810 23:39:09.479667 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40579
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab
--principal=oryx/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:09.840893 9973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:09.841202 9973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:09.841327 9973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:09.848955 9973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:09.849104 9973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:09.849232 9973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:09.859841 9973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab
--principal=oryx/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:09.861672 9973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:09.863214 9973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:09.877228 9987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:09.877161 9988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:09.877240 9991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:09.878513 9973 server_base.cc:1034] running on GCE node
I20260810 23:39:09.879201 9973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:09.880355 9973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:09.881578 9973 hybrid_clock.cc:648] HybridClock initialized: now 1786405149881495 us; error 82 us; skew 500 ppm
I20260810 23:39:09.881950 9973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:09 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405149, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:09.885466 9973 init.cc:377] Logged in from keytab as oryx/127.30.33.1@KRBTEST.COM (short username oryx)
I20260810 23:39:09.887152 9973 webserver.cc:460] Webserver started at http://127.30.33.1:38433/ using document root <none> and password file <none>
I20260810 23:39:09.887985 9973 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:09.888123 9973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:09.888453 9973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:09.892087 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "8b3c2f7a295c4fe891e0044311bc9a99"
format_stamp: "Formatted at 2026-08-10 23:39:09 on dist-test-slave-pww0"
server_key: "512ceff7fce8983f0cef889b103f3896"
server_key_iv: "18d141d917789df712627551cc8ea946"
server_key_version: "encryptionkey@0"
I20260810 23:39:09.892997 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "8b3c2f7a295c4fe891e0044311bc9a99"
format_stamp: "Formatted at 2026-08-10 23:39:09 on dist-test-slave-pww0"
server_key: "512ceff7fce8983f0cef889b103f3896"
server_key_iv: "18d141d917789df712627551cc8ea946"
server_key_version: "encryptionkey@0"
I20260810 23:39:09.898789 9973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:39:09.902889 10001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:09.904513 9973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:09.904704 9973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "8b3c2f7a295c4fe891e0044311bc9a99"
format_stamp: "Formatted at 2026-08-10 23:39:09 on dist-test-slave-pww0"
server_key: "512ceff7fce8983f0cef889b103f3896"
server_key_iv: "18d141d917789df712627551cc8ea946"
server_key_version: "encryptionkey@0"
I20260810 23:39:09.904938 9973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:09.927955 9973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:09.932439 9973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:09.932740 9973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:09.933804 9973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:09.935257 9973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:09.935339 9973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:09.935482 9973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:09.935537 9973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 23:39:09.968858 9973 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43999
I20260810 23:39:09.968915 10215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43999 every 8 connection(s)
I20260810 23:39:09.970193 9973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:09.973097 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 9973
I20260810 23:39:09.973330 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:39:09.974159 30852 external_mini_cluster.cc:1428] Setting key 7b06c5ddd6c2b21526c5a2b13a1512bc
Aug 10 23:39:09 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405149, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:09.992777 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:09.972918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:59321 (local address 127.30.33.62:40579)
0810 23:39:09.973503 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:09.973511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:09.974884 (+ 1373us) server_negotiation.cc:407] Connection header received
0810 23:39:09.977232 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:09.977238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:09.977324 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:09.977417 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:09.979091 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:09.979852 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:09.981515 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:09.981917 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:09.982035 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:09.986272 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:09.986278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:09.986283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:09.986324 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:09.989392 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:09.990239 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:09.990256 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:09.990260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:09.990447 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:09.990888 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:09.990892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:09.990894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:09.991235 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:09.991375 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:09.992182 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 23:39:09.992427 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":326,"threads_started":1}
I20260810 23:39:09.993664 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:09.973746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.1:59321)
0810 23:39:09.974687 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:09.974745 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:09.976943 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:09.977694 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:09.977703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:09.978287 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:09.978903 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:09.978927 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:09.980025 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:09.980029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:09.981342 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:09.981356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:09.982156 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:09.982164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:09.982411 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:09.983122 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:09.983146 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:09.985983 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:09.989580 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:09.989589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:09.989606 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:09.990096 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:09.990555 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:09.990558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:09.990560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:09.990782 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:09.991349 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:09.991355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:09.991777 (+ 422us) client_negotiation.cc:769] Sending connection context
0810 23:39:09.992029 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 23:39:09.992455 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":388,"threads_started":1}
I20260810 23:39:09.995297 10216 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40579
I20260810 23:39:09.995689 10216 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:09.996658 10216 heartbeater.cc:511] Master 127.30.33.62:40579 requested a full tablet report, sending...
I20260810 23:39:09.999428 9869 ts_manager.cc:194] Registered new tserver with Master: 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:10.001673 9869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:59321
WARNING: no policy specified for oryx/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab.
Entry for principal oryx/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab.
I20260810 23:39:10.039394 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40579
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab
--principal=oryx/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:10.395443 10227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:10.395947 10227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:10.396090 10227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:10.403986 10227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:10.404124 10227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:10.404264 10227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:10.414911 10227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab
--principal=oryx/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:10.416352 10227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:10.417726 10227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:10.431731 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:10.431772 10239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:10.431676 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:10.432502 10227 server_base.cc:1034] running on GCE node
I20260810 23:39:10.433171 10227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:10.434231 10227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:10.435441 10227 hybrid_clock.cc:648] HybridClock initialized: now 1786405150435388 us; error 51 us; skew 500 ppm
I20260810 23:39:10.435810 10227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:10 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405150, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:10.439103 10227 init.cc:377] Logged in from keytab as oryx/127.30.33.2@KRBTEST.COM (short username oryx)
I20260810 23:39:10.440800 10227 webserver.cc:460] Webserver started at http://127.30.33.2:37665/ using document root <none> and password file <none>
I20260810 23:39:10.441530 10227 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:10.441641 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:10.441927 10227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:10.445279 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "c6a3eb08c9b5401b99a2dc64b2590c53"
format_stamp: "Formatted at 2026-08-10 23:39:10 on dist-test-slave-pww0"
server_key: "0440649ae07245d09ce0533dc1053837"
server_key_iv: "e2840527e822630dd522f249e1a8b130"
server_key_version: "encryptionkey@0"
I20260810 23:39:10.446108 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "c6a3eb08c9b5401b99a2dc64b2590c53"
format_stamp: "Formatted at 2026-08-10 23:39:10 on dist-test-slave-pww0"
server_key: "0440649ae07245d09ce0533dc1053837"
server_key_iv: "e2840527e822630dd522f249e1a8b130"
server_key_version: "encryptionkey@0"
I20260810 23:39:10.451853 10227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:39:10.455780 10261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:10.457360 10227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:39:10.457549 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "c6a3eb08c9b5401b99a2dc64b2590c53"
format_stamp: "Formatted at 2026-08-10 23:39:10 on dist-test-slave-pww0"
server_key: "0440649ae07245d09ce0533dc1053837"
server_key_iv: "e2840527e822630dd522f249e1a8b130"
server_key_version: "encryptionkey@0"
I20260810 23:39:10.457798 10227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:10.502168 10227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:10.506924 10227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:10.507253 10227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:10.508400 10227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:10.509964 10227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:10.510043 10227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:10.510156 10227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:10.510216 10227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:10.542670 10227 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:41157
I20260810 23:39:10.542732 10478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:41157 every 8 connection(s)
I20260810 23:39:10.544130 10227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:10.548622 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 10227
I20260810 23:39:10.548828 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:39:10.549292 30852 external_mini_cluster.cc:1428] Setting key 2e6a4eb0ca586ffab6ca7917eb2f121d
Aug 10 23:39:10 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405150, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:10.566584 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:10.547010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:56275 (local address 127.30.33.62:40579)
0810 23:39:10.547542 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:10.547547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:10.548617 (+ 1070us) server_negotiation.cc:407] Connection header received
0810 23:39:10.550884 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:10.550888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:10.550956 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:10.551052 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:10.552940 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:10.553718 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:10.555416 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:10.555907 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:10.556028 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:10.560085 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:10.560091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:10.560094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:10.560135 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:10.563112 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:10.564051 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:10.564057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:10.564059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:10.564213 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:10.564756 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:10.564760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:10.564762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:10.565057 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:10.565175 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:10.566079 (+ 904us) server_negotiation.cc:299] Negotiation successful
0810 23:39:10.566292 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260810 23:39:10.568306 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:10.547643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.2:56275)
0810 23:39:10.548428 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:10.548479 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:10.550535 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:10.551375 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:10.551388 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:10.552080 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:10.552722 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:10.552742 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:10.553902 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:10.553906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:10.555238 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:10.555255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:10.556165 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:10.556173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:10.556423 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:10.557201 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:10.557227 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:10.559799 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:10.563308 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:10.563316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:10.563333 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:10.563895 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:10.564342 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:10.564346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:10.564350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:10.564632 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:10.565191 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:10.565199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:10.565680 (+ 481us) client_negotiation.cc:769] Sending connection context
0810 23:39:10.566003 (+ 323us) client_negotiation.cc:240] Negotiation successful
0810 23:39:10.566775 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":284,"threads_started":1}
I20260810 23:39:10.570042 10479 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40579
I20260810 23:39:10.570508 10479 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:10.571502 10479 heartbeater.cc:511] Master 127.30.33.62:40579 requested a full tablet report, sending...
I20260810 23:39:10.573534 9869 ts_manager.cc:194] Registered new tserver with Master: c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157)
I20260810 23:39:10.574718 9869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:56275
WARNING: no policy specified for oryx/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab.
Entry for principal oryx/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab.
I20260810 23:39:10.615979 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40579
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab
--principal=oryx/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:10.966645 10499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:10.966966 10499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:10.967062 10499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:10.977246 10499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:10.977377 10499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:10.977495 10499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:10.987741 10499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab
--principal=oryx/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:10.989091 10499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:10.990464 10499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:11.004529 10506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:11.004443 10510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:11.004482 10504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:11.005425 10216 heartbeater.cc:503] Master 127.30.33.62:40579 was elected leader, sending a full tablet report...
I20260810 23:39:11.005424 10499 server_base.cc:1034] running on GCE node
I20260810 23:39:11.006278 10499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:11.007450 10499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:11.008721 10499 hybrid_clock.cc:648] HybridClock initialized: now 1786405151008660 us; error 55 us; skew 500 ppm
I20260810 23:39:11.009094 10499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405151, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:11.012768 10499 init.cc:377] Logged in from keytab as oryx/127.30.33.3@KRBTEST.COM (short username oryx)
I20260810 23:39:11.014503 10499 webserver.cc:460] Webserver started at http://127.30.33.3:36827/ using document root <none> and password file <none>
I20260810 23:39:11.015292 10499 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:11.015419 10499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:11.015801 10499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:11.019069 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "8a41d48a84aa4d34a000ba1a59f213c3"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "e14c73c2fc1f4ec5bb928d62d34dcdf0"
server_key_iv: "b75a11a97a8bd3b3d76c95cdf836425f"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.020000 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "8a41d48a84aa4d34a000ba1a59f213c3"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "e14c73c2fc1f4ec5bb928d62d34dcdf0"
server_key_iv: "b75a11a97a8bd3b3d76c95cdf836425f"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.025734 10499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 23:39:11.029798 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:11.031410 10499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:39:11.031662 10499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "8a41d48a84aa4d34a000ba1a59f213c3"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "e14c73c2fc1f4ec5bb928d62d34dcdf0"
server_key_iv: "b75a11a97a8bd3b3d76c95cdf836425f"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.031874 10499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:11.044423 10499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:11.049103 10499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:11.049407 10499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:11.050464 10499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:11.051997 10499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:11.052076 10499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:11.052194 10499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:11.052245 10499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:11.083945 10499 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:44785
I20260810 23:39:11.084035 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:44785 every 8 connection(s)
I20260810 23:39:11.085340 10499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:11.094681 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 10499
I20260810 23:39:11.094893 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:39:11.095481 30852 external_mini_cluster.cc:1428] Setting key cb6659e8d63564ef91b8a748f967e7da
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405151, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:11.108640 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.088199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:53255 (local address 127.30.33.62:40579)
0810 23:39:11.088699 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.088703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.089777 (+ 1074us) server_negotiation.cc:407] Connection header received
0810 23:39:11.091653 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.091658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.091746 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.091838 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.093428 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.094288 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.096617 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.097213 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.097319 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.101326 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.101332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.101335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.101372 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.104638 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.105699 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.105703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.105705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.105842 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.106363 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.106368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.106370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.106712 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.106858 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.107841 (+ 983us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.108138 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":233,"threads_started":1}
I20260810 23:39:11.109802 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.088826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.3:53255)
0810 23:39:11.089600 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:11.089641 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:11.091306 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:11.092046 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:11.092055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:11.092623 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:11.093236 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.093256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.094493 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.094500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.096443 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.096467 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.097482 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.097492 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.097792 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.098614 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:11.098650 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:11.101108 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:11.104823 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.104834 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.104854 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.105545 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.105979 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.105983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.105986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.106216 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.106869 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:11.106877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:11.107289 (+ 412us) client_negotiation.cc:769] Sending connection context
0810 23:39:11.107679 (+ 390us) client_negotiation.cc:240] Negotiation successful
0810 23:39:11.108315 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":304,"threads_started":1}
I20260810 23:39:11.111702 10729 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40579
I20260810 23:39:11.112051 10729 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:11.112816 10729 heartbeater.cc:511] Master 127.30.33.62:40579 requested a full tablet report, sending...
I20260810 23:39:11.114372 9869 ts_manager.cc:194] Registered new tserver with Master: 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785)
I20260810 23:39:11.115206 9869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:53255
I20260810 23:39:11.124478 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.61.keytab.
Entry for principal oryx/127.30.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.61.keytab.
I20260810 23:39:11.191912 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:41497
--webserver_interface=127.30.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.61.keytab
--principal=oryx/127.30.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:40579,127.30.33.61:41497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:11.537719 10739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:11.538031 10739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:11.538160 10739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:11.545648 10739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:11.545792 10739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:11.545871 10739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:11.545948 10739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:11.546008 10739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:11.556306 10739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:40579,127.30.33.61:41497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.61.keytab
--principal=oryx/127.30.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:41497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
W20260810 23:39:11.556742 10739 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.33.62:40579,127.30.33.61:41497'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 23:39:11.557912 10739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:11.559305 10739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:11.571408 10746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:11.571429 10745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:11.572136 10749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:11.572647 10739 server_base.cc:1034] running on GCE node
I20260810 23:39:11.573534 10739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:11.575054 10739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:11.576294 10739 hybrid_clock.cc:648] HybridClock initialized: now 1786405151576210 us; error 60 us; skew 500 ppm
I20260810 23:39:11.576651 10739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:11.578395 10479 heartbeater.cc:503] Master 127.30.33.62:40579 was elected leader, sending a full tablet report...
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405151, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:11.580216 10739 init.cc:377] Logged in from keytab as oryx/127.30.33.61@KRBTEST.COM (short username oryx)
I20260810 23:39:11.581795 10739 webserver.cc:460] Webserver started at http://127.30.33.61:46309/ using document root <none> and password file <none>
I20260810 23:39:11.582507 10739 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:11.582607 10739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:11.582890 10739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:11.586125 10739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/instance:
uuid: "c8029edc52e745609f8f4878a3704ee5"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "d0f3fdbd88137638300004eed73f8bab"
server_key_iv: "de13b5017e795f92c139f1a4a57ef29a"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.586928 10739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal/instance:
uuid: "c8029edc52e745609f8f4878a3704ee5"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "d0f3fdbd88137638300004eed73f8bab"
server_key_iv: "de13b5017e795f92c139f1a4a57ef29a"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.592538 10739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:39:11.596524 10761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:11.598073 10739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:39:11.598274 10739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
uuid: "c8029edc52e745609f8f4878a3704ee5"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "d0f3fdbd88137638300004eed73f8bab"
server_key_iv: "de13b5017e795f92c139f1a4a57ef29a"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.598503 10739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:11.620466 10739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:11.625172 10739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:11.625452 10739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:11.644250 10739 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.61:41497
I20260810 23:39:11.644275 10860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.61:41497 every 8 connection(s)
I20260810 23:39:11.645751 10739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
I20260810 23:39:11.649611 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 10739
I20260810 23:39:11.650102 10861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:11.660620 10861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.61@KRBTEST.COM
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405151, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.61@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:11.677428 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.652145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.30.33.61:41497)
0810 23:39:11.653030 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.653048 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.653099 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:39:11.655219 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.655250 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.655819 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.656233 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.658371 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.659848 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.661897 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.662809 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.662988 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.668412 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.668426 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.668444 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.668505 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.672132 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.673135 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.673145 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.673156 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.673390 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.673903 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.673907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.673909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.674569 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.674891 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.675370 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.675887 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":429,"threads_started":1}
I20260810 23:39:11.681396 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.664200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.30.33.62:40579)
0810 23:39:11.664772 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.664778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.665646 (+ 868us) server_negotiation.cc:407] Connection header received
0810 23:39:11.665867 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.665871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.665942 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.666024 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.667420 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.668315 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.669879 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.670536 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.670693 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.674617 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.674625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.674630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.674686 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.678071 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.679053 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.679057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.679059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.679194 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.679760 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.679765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.679767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.680158 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.680291 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.680902 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.681121 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":274,"threads_started":1}
I20260810 23:39:11.681413 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.664197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:43814)
0810 23:39:11.665450 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:11.665519 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:11.665730 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:11.666137 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:11.666145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:11.666950 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:11.667230 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.667244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.668487 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.668491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.669710 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.669724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.670759 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.670764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.670907 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.671600 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:11.671620 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:11.674363 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:11.678213 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.678224 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.678232 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.678861 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.679324 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.679330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.679333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.679607 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.680298 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:11.680305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:11.680530 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 23:39:11.680749 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 23:39:11.681121 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":754,"threads_started":1}
I20260810 23:39:11.683300 10861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405151, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.61@KRBTEST.COM for oryx/127.30.33.61@KRBTEST.COM
I20260810 23:39:11.696475 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.684027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.0.0.1:59010)
0810 23:39:11.684228 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:11.684245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:11.684347 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:11.684641 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:11.684644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:11.684986 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:11.685195 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.685206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.686236 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.686241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.687235 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.687249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.688083 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.688088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.688242 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.688837 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:11.688857 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:11.691040 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:11.694041 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.694046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.694048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.694453 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.694818 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.694822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.694824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.695037 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.695608 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:11.695612 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:11.695783 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 23:39:11.695913 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 23:39:11.696184 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 23:39:11.696657 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.684062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.30.33.61:41497)
0810 23:39:11.684221 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.684225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.684330 (+ 105us) server_negotiation.cc:407] Connection header received
0810 23:39:11.684434 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.684437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.684498 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.684592 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.685346 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.686052 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.687409 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.687920 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.688034 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.691253 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.691258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.691262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.691298 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.693886 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.694566 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.694572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.694575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.694696 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.695144 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.695147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.695149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.695443 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.695621 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.696001 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.696333 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":60}
I20260810 23:39:11.701709 10861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Bootstrap starting.
I20260810 23:39:11.705297 10861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:11.706389 10861 log.cc:824] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:11.709514 10861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: No bootstrap required, opened a new log
I20260810 23:39:11.716847 10861 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } }
I20260810 23:39:11.717236 10861 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:11.717314 10861 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8029edc52e745609f8f4878a3704ee5, State: Initialized, Role: FOLLOWER
I20260810 23:39:11.717931 10861 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } }
I20260810 23:39:11.718824 10879 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } } }
I20260810 23:39:11.719151 10879 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:39:11.719923 10861 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:39:11.729094 10894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:39:11.729272 10894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:39:11.732594 10739 master_runner.cc:428] Detected that this master c8029edc52e745609f8f4878a3704ee5 is joining an existing cluster
I20260810 23:39:11.732717 10739 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.33.61:41497
WARNING: no policy specified for oryx/127.30.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.60@KRBTEST.COM" created.
I20260810 23:39:11.765044 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.747089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:35770)
0810 23:39:11.747993 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:11.748015 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:11.748298 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:11.749092 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:11.749097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:11.749864 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:11.750279 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.750300 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.751820 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.751825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.753405 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.753425 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.754716 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.754723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.754902 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.755704 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:11.755727 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:11.757143 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:11.761489 (+ 4346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.761494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.761497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.762084 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.762693 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.762707 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.762712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.762901 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.763658 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:11.763664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:11.763907 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 23:39:11.764108 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 23:39:11.764519 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":1,"thread_start_us":550,"threads_started":1}
I20260810 23:39:11.765518 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.747096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.30.33.62:40579)
0810 23:39:11.747311 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.747318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.748133 (+ 815us) server_negotiation.cc:407] Connection header received
0810 23:39:11.748498 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.748506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.748619 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.748699 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.750494 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.751610 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.753604 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.754200 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.754325 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.757417 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.757425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.757430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.757478 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.761353 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.762316 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.762322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.762326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.762495 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.763057 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.763062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.763067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.763501 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.763662 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.764421 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.764820 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:39:11.774092 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.748639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.30.33.61:41497)
0810 23:39:11.749023 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.749032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.757789 (+ 8757us) server_negotiation.cc:407] Connection header received
0810 23:39:11.758083 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.758089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.758159 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.758224 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.759749 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.760945 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.763871 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.764537 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.764683 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.768019 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.768025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.768029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.768060 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.771444 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.772184 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.772189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.772192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.772332 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.772666 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.772670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.772673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.772995 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.773133 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.773552 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.773765 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260810 23:39:11.775108 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.748841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.0.0.1:54710)
0810 23:39:11.757654 (+ 8813us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:11.757711 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:11.757957 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:11.758426 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:11.758431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:11.759118 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:11.759571 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.759588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.761265 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.761270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.763730 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.763744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.765841 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.765848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.766021 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.766673 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:11.766696 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:11.767827 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:11.771622 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.771626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.771629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.772064 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.772447 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.772451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.772454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.772579 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.773204 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:11.773207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:11.773334 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 23:39:11.774436 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0810 23:39:11.774733 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8635,"thread_start_us":6532,"threads_started":1}
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405149, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.62@KRBTEST.COM for oryx/127.30.33.61@KRBTEST.COM
I20260810 23:39:11.791047 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.773251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.30.33.61:41497)
0810 23:39:11.774404 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.774409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.777512 (+ 3103us) server_negotiation.cc:407] Connection header received
0810 23:39:11.777879 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.777885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.777952 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.778035 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.779208 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.779979 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.781296 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.781766 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.781878 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.785270 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.785276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.785279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.785313 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.788062 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.788850 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.788857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.788859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.789011 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.789397 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.789401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.789403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.789737 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.789874 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.790393 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.790638 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":4443,"threads_started":1}
I20260810 23:39:11.791090 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.772923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.0.0.1:54712)
0810 23:39:11.777355 (+ 4432us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:11.777415 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:11.777680 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:11.778115 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:11.778123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:11.778818 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:11.779052 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.779062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.780109 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.780115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.781149 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.781162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.781928 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.781933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.782135 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.782769 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:11.782786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:11.785073 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:11.788231 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.788249 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.788258 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.788723 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.789125 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.789131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.789135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.789289 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.789878 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:11.789888 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:11.790087 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 23:39:11.790283 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 23:39:11.790653 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4185,"thread_start_us":4031,"threads_started":1}
I20260810 23:39:11.793606 9869 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.33.61:41497: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c8029edc52e745609f8f4878a3704ee5"
member_type: NON_VOTER
last_known_addr {
host: "127.30.33.61"
port: 41497
}
attrs {
promote: true
}
}
dest_uuid: "97a992b3716d4a66b6ca82dd811968fd"
cas_config_opid_index: -1
I20260810 23:39:11.795009 9869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } }
I20260810 23:39:11.801210 10922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8029edc52e745609f8f4878a3704ee5 (127.30.33.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } }
I20260810 23:39:11.802067 10806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.60.keytab.
Entry for principal oryx/127.30.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.60.keytab.
I20260810 23:39:11.806644 10806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Refusing update from remote peer 97a992b3716d4a66b6ca82dd811968fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 23:39:11.806890 10922 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.33.61:41497
I20260810 23:39:11.807758 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:39057
--webserver_interface=127.30.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.60.keytab
--principal=oryx/127.30.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:39:11.808939 10739 master.cc:537] Master@127.30.33.61:41497 shutting down...
I20260810 23:39:11.810007 10922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:11.811992 10923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } }
I20260810 23:39:11.812371 10923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
W20260810 23:39:11.813411 9845 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 23:39:11.824113 10739 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:11.824795 10739 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:11.825136 10739 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: stopping tablet replica
W20260810 23:39:11.825613 9845 consensus_peers.cc:487] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd -> Peer c8029edc52e745609f8f4878a3704ee5 (127.30.33.61:41497): Couldn't send request to peer c8029edc52e745609f8f4878a3704ee5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:39:11.832911 10739 master.cc:552] Master@127.30.33.61:41497 shutdown complete.
I20260810 23:39:11.833061 10739 master_runner.cc:305] Clearing existing system tablet
I20260810 23:39:11.833549 10739 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:11.833658 10739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:11.836930 10739 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.004s
I20260810 23:39:11.839560 10929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:11.840054 10739 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:11.840371 10739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 23:39:11.840468 10739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
uuid: "c8029edc52e745609f8f4878a3704ee5"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "d0f3fdbd88137638300004eed73f8bab"
server_key_iv: "de13b5017e795f92c139f1a4a57ef29a"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.846210 10739 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:11.853662 10739 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:39:11.853902 10739 log.cc:1196] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 23:39:11.854406 10739 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Deleting consensus metadata
I20260810 23:39:11.854682 10739 master_runner.cc:315] Copying system tablet from 127.30.33.62:40579
I20260810 23:39:11.857810 10739 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.33.62:40579
I20260810 23:39:11.875052 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.859016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.30.33.62:40579)
0810 23:39:11.859212 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:11.859218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:11.859767 (+ 549us) server_negotiation.cc:407] Connection header received
0810 23:39:11.860033 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:11.860039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:11.860133 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:11.860256 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:11.861462 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.862654 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.864621 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.865319 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.865466 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.867950 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:11.867957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:11.867962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:11.868005 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:11.871683 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.872639 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.872647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.872650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.872862 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:11.873332 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:11.873337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:11.873340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:11.873730 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:11.873895 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:11.874409 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 23:39:11.874680 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:39:11.875102 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:11.858940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:35780)
0810 23:39:11.859625 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:11.859646 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:11.859825 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:11.860300 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:11.860309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:11.860851 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:11.861290 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.861308 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.862817 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.862823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.864430 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:11.864450 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:11.865515 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:11.865523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:11.865719 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:11.866441 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:11.866467 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:11.867681 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:11.871879 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.871886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.871891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.872456 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.873001 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:11.873007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:11.873010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:11.873192 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:11.873872 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:11.873878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:11.874088 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 23:39:11.874266 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 23:39:11.874624 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":388,"threads_started":1}
I20260810 23:39:11.876585 9908 tablet_copy_service.cc:140] P 97a992b3716d4a66b6ca82dd811968fd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c8029edc52e745609f8f4878a3704ee5 ({username='oryx', principal='oryx/127.30.33.61@KRBTEST.COM'} at 127.0.0.1:35780)
I20260810 23:39:11.877035 9908 tablet_copy_service.cc:161] P 97a992b3716d4a66b6ca82dd811968fd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c8029edc52e745609f8f4878a3704ee5 at {username='oryx', principal='oryx/127.30.33.61@KRBTEST.COM'} at 127.0.0.1:35780: session id = c8029edc52e745609f8f4878a3704ee5-00000000000000000000000000000000
I20260810 23:39:11.882586 9908 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:39:11.885963 10739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:39:11.893570 10739 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: tablet copy: Starting download of 0 data blocks...
I20260810 23:39:11.894048 10739 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: tablet copy: Starting download of 1 WAL segments...
I20260810 23:39:11.898274 10739 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:39:11.903421 9908 tablet_copy_service.cc:342] P 97a992b3716d4a66b6ca82dd811968fd: Request end of tablet copy session c8029edc52e745609f8f4878a3704ee5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.33.61@KRBTEST.COM'} at 127.0.0.1:35780
I20260810 23:39:11.903807 9908 tablet_copy_service.cc:433] P 97a992b3716d4a66b6ca82dd811968fd: ending tablet copy session c8029edc52e745609f8f4878a3704ee5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c8029edc52e745609f8f4878a3704ee5
I20260810 23:39:11.917619 10739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:11.920612 10940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:11.921036 10944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:11.925902 10739 server_base.cc:1034] running on GCE node
W20260810 23:39:11.927848 10947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:11.928397 10739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:11.929169 10739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:11.930411 10739 hybrid_clock.cc:648] HybridClock initialized: now 1786405151930331 us; error 52 us; skew 500 ppm
I20260810 23:39:11.930656 10739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:11 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405151, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:11.932929 10739 init.cc:377] Logged in from keytab as oryx/127.30.33.61@KRBTEST.COM (short username oryx)
I20260810 23:39:11.934306 10739 webserver.cc:460] Webserver started at http://127.30.33.61:32997/ using document root <none> and password file <none>
I20260810 23:39:11.934702 10739 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:11.934785 10739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:11.938165 10739 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:39:11.940296 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:11.940828 10739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:39:11.940996 10739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
uuid: "c8029edc52e745609f8f4878a3704ee5"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "d0f3fdbd88137638300004eed73f8bab"
server_key_iv: "de13b5017e795f92c139f1a4a57ef29a"
server_key_version: "encryptionkey@0"
I20260810 23:39:11.941180 10739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:11.953040 10739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:11.957499 10739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:11.996381 10739 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.61:41497
I20260810 23:39:11.996443 11059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.61:41497 every 8 connection(s)
I20260810 23:39:11.997479 10739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
I20260810 23:39:12.003168 11061 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:39:12.007536 11061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Bootstrap starting.
I20260810 23:39:12.029884 11061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:39:12.030689 11061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: Bootstrap complete.
I20260810 23:39:12.032166 11061 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } }
I20260810 23:39:12.032581 11061 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8029edc52e745609f8f4878a3704ee5, State: Initialized, Role: LEARNER
I20260810 23:39:12.033008 11061 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } }
I20260810 23:39:12.034060 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } }
I20260810 23:39:12.034582 11061 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:12.034590 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:39:12.046418 11083 catalog_manager.cc:1259] Loaded cluster ID: 97c24683486b4c94963bd5fc148581c3
I20260810 23:39:12.046552 11083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: loading cluster ID for follower catalog manager: success
I20260810 23:39:12.050385 11083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: acquiring CA information for follower catalog manager: success
I20260810 23:39:12.053653 11083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:39:12.117926 10729 heartbeater.cc:503] Master 127.30.33.62:40579 was elected leader, sending a full tablet report...
I20260810 23:39:12.218195 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.193292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.0.0.1:54720)
0810 23:39:12.193747 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.193770 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.193909 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.210467 (+ 16558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.210473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.210512 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:12.210735 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.210747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.213134 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.213141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.215285 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.215311 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.217319 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.217328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.217578 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.217666 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.217750 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.217918 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260810 23:39:12.219671 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.194295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.30.33.61:41497)
0810 23:39:12.209930 (+ 15635us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.209937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.209971 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:39:12.210053 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.210060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.210161 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.210891 (+ 730us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:12.211167 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.212878 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.215574 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.217070 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.218787 (+ 1717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.219063 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.219127 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.219284 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15446,"thread_start_us":15235,"threads_started":1}
I20260810 23:39:12.220968 11008 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 23:39:12.221911 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97a992b3716d4a66b6ca82dd811968fd. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } }
I20260810 23:39:12.222214 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: LEARNER
W20260810 23:39:12.257087 10927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:12.257439 10927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:12.257612 10927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:12.266079 10927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:12.266223 10927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:12.266289 10927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:12.266371 10927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:12.266438 10927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:12.277523 10927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.60.keytab
--principal=oryx/127.30.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:12.278999 10927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:12.280400 10927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:12.295176 11093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:12.295146 11094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:12.299182 11097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:12.301144 10927 server_base.cc:1034] running on GCE node
I20260810 23:39:12.302296 10927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:12.304484 10927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:12.305747 10927 hybrid_clock.cc:648] HybridClock initialized: now 1786405152305678 us; error 63 us; skew 500 ppm
I20260810 23:39:12.306201 10927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405152, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:12.310798 10927 init.cc:377] Logged in from keytab as oryx/127.30.33.60@KRBTEST.COM (short username oryx)
I20260810 23:39:12.316198 10927 webserver.cc:460] Webserver started at http://127.30.33.60:41373/ using document root <none> and password file <none>
I20260810 23:39:12.317147 10927 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:12.317299 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:12.317641 10927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:12.321861 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/instance:
uuid: "d8babcf8bb3647998cf0020b94c67171"
format_stamp: "Formatted at 2026-08-10 23:39:12 on dist-test-slave-pww0"
server_key: "42a67c7a3f3d4c0c337772d4de882c98"
server_key_iv: "f68f3fab6504d0d8ad5c11c48c243251"
server_key_version: "encryptionkey@0"
I20260810 23:39:12.322881 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal/instance:
uuid: "d8babcf8bb3647998cf0020b94c67171"
format_stamp: "Formatted at 2026-08-10 23:39:12 on dist-test-slave-pww0"
server_key: "42a67c7a3f3d4c0c337772d4de882c98"
server_key_iv: "f68f3fab6504d0d8ad5c11c48c243251"
server_key_version: "encryptionkey@0"
I20260810 23:39:12.329916 10927 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260810 23:39:12.333917 11114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:12.335942 10927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260810 23:39:12.336151 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
uuid: "d8babcf8bb3647998cf0020b94c67171"
format_stamp: "Formatted at 2026-08-10 23:39:12 on dist-test-slave-pww0"
server_key: "42a67c7a3f3d4c0c337772d4de882c98"
server_key_iv: "f68f3fab6504d0d8ad5c11c48c243251"
server_key_version: "encryptionkey@0"
I20260810 23:39:12.336364 10927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:12.350385 10927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:12.355683 10927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:12.356014 10927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:12.377408 10927 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.60:39057
I20260810 23:39:12.377424 11225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.60:39057 every 8 connection(s)
I20260810 23:39:12.379145 10927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
I20260810 23:39:12.382222 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 10927
I20260810 23:39:12.383445 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:12.394636 11226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.60@KRBTEST.COM
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405152, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.60@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:12.413914 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.397983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.30.33.62:40579)
0810 23:39:12.399519 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.399523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.399541 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:39:12.399593 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.399597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.399671 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.399787 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.400957 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.401813 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.403185 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.403851 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.403984 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.407773 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.407779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.407782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.407819 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.410713 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.411654 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.411659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.411661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.411798 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.412176 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.412178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.412180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.412439 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.412586 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.413448 (+ 862us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.413636 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":1303,"threads_started":1}
I20260810 23:39:12.415195 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.397890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:35792)
0810 23:39:12.398776 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.398832 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.399022 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.399857 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.399863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.400516 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.400783 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.400793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.401962 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.401966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.403016 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.403029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.404104 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.404109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.404267 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.404901 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.404928 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.407533 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.410864 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.410879 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.410894 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.411513 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.411910 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.411914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.411917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.412079 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.412584 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.412593 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.412815 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.413186 (+ 371us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.413720 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":151,"spinlock_wait_cycles":26496,"thread_start_us":358,"threads_started":1}
I20260810 23:39:12.415215 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.384967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.30.33.60:39057)
0810 23:39:12.392596 (+ 7629us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.392610 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.392660 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:39:12.394468 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.394495 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.395000 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.395399 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.397445 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.399540 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.400911 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.401583 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.401741 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.406581 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.406590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.406606 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.406646 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.409960 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.410927 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.410936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.410939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.411178 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.411646 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.411650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.411652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.412378 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.412652 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.413054 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.413601 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":7383,"thread_start_us":6241,"threads_started":1}
I20260810 23:39:12.418015 11226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405152, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.60@KRBTEST.COM for oryx/127.30.33.61@KRBTEST.COM
I20260810 23:39:12.431277 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.418904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.30.33.61:41497)
0810 23:39:12.419096 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.419101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.419116 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:39:12.419162 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.419166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.419226 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.419316 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.420070 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.420804 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.422054 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.422466 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.422582 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.425831 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.425837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.425841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.425883 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.428570 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.429310 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.429314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.429317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.429459 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.429868 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.429874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.429876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.430168 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.430323 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.430694 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.430933 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:39:12.431305 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.418699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.0.0.1:54730)
0810 23:39:12.418866 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.418882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.418981 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.419355 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.419359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.419775 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.419922 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.419930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.420950 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.420954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.421912 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.421924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.422622 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.422628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.422823 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.423421 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.423438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.425611 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.428725 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.428731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.428736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.429174 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.429571 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.429576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.429580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.429753 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.430304 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.430309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.430458 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.430581 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.430916 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 23:39:12.432418 11226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405152, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.60@KRBTEST.COM for oryx/127.30.33.60@KRBTEST.COM
I20260810 23:39:12.445581 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.433249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49820)
0810 23:39:12.433429 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.433448 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.433554 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.433815 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.433819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.434190 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.434342 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.434350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.435344 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.435349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.436321 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.436334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.437008 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.437013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.437225 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.437828 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.437845 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.439956 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.443146 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.443160 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.443163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.443571 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.443945 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.443950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.443954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.444103 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.444600 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.444604 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.444746 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.444869 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.445168 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:39:12.445603 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.433292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.30.33.60:39057)
0810 23:39:12.433429 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.433435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.433505 (+ 70us) server_negotiation.cc:407] Connection header received
0810 23:39:12.433647 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.433651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.433703 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.433786 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.434466 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.435228 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.436446 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.436857 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.437001 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.440151 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.440155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.440158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.440189 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.442994 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.443714 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.443718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.443720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.443843 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.444212 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.444215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.444217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.444493 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.444626 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.444990 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.445314 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":42}
I20260810 23:39:12.451169 11226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Bootstrap starting.
I20260810 23:39:12.454808 11226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:12.456116 11226 log.cc:824] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:12.459333 11226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: No bootstrap required, opened a new log
I20260810 23:39:12.467367 11226 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } }
I20260810 23:39:12.467845 11226 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:12.467976 11226 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8babcf8bb3647998cf0020b94c67171, State: Initialized, Role: FOLLOWER
I20260810 23:39:12.468660 11226 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } }
I20260810 23:39:12.469779 11246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } } }
I20260810 23:39:12.470211 11246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:39:12.471123 11226 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:39:12.481254 11269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:39:12.481424 11269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:39:12.485479 10927 master_runner.cc:428] Detected that this master d8babcf8bb3647998cf0020b94c67171 is joining an existing cluster
I20260810 23:39:12.485653 10927 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.33.60:39057
I20260810 23:39:12.507786 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.490666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.30.33.60:39057)
0810 23:39:12.490885 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.490891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.491616 (+ 725us) server_negotiation.cc:407] Connection header received
0810 23:39:12.491884 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.491890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.491963 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.492036 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.493631 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.494777 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.496837 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.497529 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.497648 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.500410 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.500416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.500420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.500455 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.504365 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.505297 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.505303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.505306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.505508 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.505960 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.505964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.505967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.506410 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.506626 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.507080 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.507341 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:39:12.509133 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.490648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49836)
0810 23:39:12.491486 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.491506 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.491732 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.492235 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.492240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.493033 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.493437 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.493454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.495016 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.495022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.496668 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.496686 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.497994 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.498001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.498167 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.498911 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.498955 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.500220 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.504542 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.504546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.504560 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.505147 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.505652 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.505657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.505661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.505842 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.506617 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.506622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.506792 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.508181 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.508563 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":13377,"threads_started":1}
I20260810 23:39:12.519686 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.492238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.30.33.61:41497)
0810 23:39:12.492437 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.492443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.505334 (+ 12891us) server_negotiation.cc:407] Connection header received
0810 23:39:12.505604 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.505610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.505700 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.505782 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.507546 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.508618 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.510696 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.511381 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.511514 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.513647 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.513652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.513655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.513689 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.516528 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.517217 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.517232 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.517236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.517396 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.517917 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.517923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.517926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.518281 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.518415 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.519026 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.519304 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:39:12.519878 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.491657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.0.0.1:54732)
0810 23:39:12.505185 (+ 13528us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.505208 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.505447 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.506017 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.506021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.506928 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.507311 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.507324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.508826 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.508831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.510511 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.510527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.511679 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.511686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.511815 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.512392 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.512418 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.513478 (+ 1060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.516732 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.516736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.516739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.517087 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.517524 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.517530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.517533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.517755 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.518397 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.518401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.518655 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.518860 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.519200 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13400,"thread_start_us":11930,"threads_started":1}
I20260810 23:39:12.520515 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.490971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.30.33.62:40579)
0810 23:39:12.491186 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.491192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.503540 (+ 12348us) server_negotiation.cc:407] Connection header received
0810 23:39:12.504068 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.504076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.504180 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.504269 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.506373 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.507498 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.509907 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.510533 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.510639 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.513504 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.513513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.513519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.513566 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.517199 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.517862 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.517868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.517872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.518043 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.518518 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.518523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.518526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.518905 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.519042 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.519669 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.520198 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:39:12.521754 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.490648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:35794)
0810 23:39:12.503361 (+ 12713us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.503386 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.503880 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.504515 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.504520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.505698 (+ 1178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.506153 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.506165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.507692 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.507697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.509735 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.509749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.510978 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.510985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.511155 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.512036 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.512064 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.513259 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.517335 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.517340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.517343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.517746 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.518170 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.518175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.518179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.518348 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.519101 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.519105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.519332 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.520784 (+ 1452us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.521121 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12490,"mutex_wait_us":30,"thread_start_us":449,"threads_started":1}
I20260810 23:39:12.522046 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 9973
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.1@KRBTEST.COM
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405149, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.62@KRBTEST.COM for oryx/127.30.33.60@KRBTEST.COM
I20260810 23:39:12.535873 11246 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:12.536271 11246 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } }
I20260810 23:39:12.545482 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.523708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.30.33.1:43999)
0810 23:39:12.524452 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.524460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.524487 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:39:12.524565 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.524574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.524863 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.525044 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.526492 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.527605 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.529358 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.530131 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.530875 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.536740 (+ 5865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.536764 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.536785 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.536838 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.540712 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.541934 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.541944 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.541948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.542143 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.542695 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.542701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.542704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.543185 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.543563 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.544681 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.545011 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":377,"threads_started":1}
I20260810 23:39:12.547199 10208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:12.547416 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.528173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.30.33.60:39057)
0810 23:39:12.528371 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.528377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.528396 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:12.528503 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.528508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.528642 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.528747 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.529762 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.530812 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.532157 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.532652 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.532775 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.536661 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.536667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.536671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.536714 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.540711 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.541662 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.541669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.541673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.541849 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.545537 (+ 3688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.545543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.545547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.546089 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.546282 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.546773 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.547036 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:39:12.551843 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.527318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49844)
0810 23:39:12.527656 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.527675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.527781 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.528820 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.528826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.529348 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.529579 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.529594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.530987 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.530993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.532010 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.532025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.532814 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.532819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.532977 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.533496 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.533514 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.536414 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.540933 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.540939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.540943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.541527 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.542002 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.542007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.542011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.545376 (+ 3365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.546260 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.546266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.546472 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.546604 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.551297 (+ 4693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260810 23:39:12.554106 9869 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.33.60:39057: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d8babcf8bb3647998cf0020b94c67171"
member_type: NON_VOTER
last_known_addr {
host: "127.30.33.60"
port: 39057
}
attrs {
promote: true
}
}
dest_uuid: "97a992b3716d4a66b6ca82dd811968fd"
cas_config_opid_index: 5
I20260810 23:39:12.555294 9869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } }
I20260810 23:39:12.557525 10923 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8babcf8bb3647998cf0020b94c67171 (127.30.33.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.560113 11246 leader_election.cc:288] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579), c8029edc52e745609f8f4878a3704ee5 (127.30.33.61:41497)
I20260810 23:39:12.561153 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8babcf8bb3647998cf0020b94c67171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8029edc52e745609f8f4878a3704ee5" is_pre_election: true
I20260810 23:39:12.562006 11007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 LEARNER]: Refusing update from remote peer 97a992b3716d4a66b6ca82dd811968fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:39:12.562073 9893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8babcf8bb3647998cf0020b94c67171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a992b3716d4a66b6ca82dd811968fd" is_pre_election: true
I20260810 23:39:12.563864 11171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:12.568652 10923 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.33.60:39057
I20260810 23:39:12.569480 10927 master.cc:537] Master@127.30.33.60:39057 shutting down...
I20260810 23:39:12.570830 11118 leader_election.cc:302] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8babcf8bb3647998cf0020b94c67171; no voters: 97a992b3716d4a66b6ca82dd811968fd, c8029edc52e745609f8f4878a3704ee5
I20260810 23:39:12.570833 11293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 23:39:12.571105 11171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Refusing update from remote peer 97a992b3716d4a66b6ca82dd811968fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:39:12.571789 11246 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:39:12.574215 11007 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8babcf8bb3647998cf0020b94c67171 (127.30.33.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.580447 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.580801 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:39:12.585911 10927 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:12.586436 10927 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:12.586661 10927 tablet_replica.cc:330] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: stopping tablet replica
I20260810 23:39:12.587168 11297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:12.595304 10927 master.cc:552] Master@127.30.33.60:39057 shutdown complete.
I20260810 23:39:12.595412 10927 master_runner.cc:305] Clearing existing system tablet
I20260810 23:39:12.595845 10927 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:12.595975 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:12.602955 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.603410 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:12.604609 11297 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: attempting to promote NON_VOTER c8029edc52e745609f8f4878a3704ee5 to VOTER
I20260810 23:39:12.605535 10927 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.005s sys 0.004s
I20260810 23:39:12.605603 11297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } }
I20260810 23:39:12.607611 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.607118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49858)
0810 23:39:12.607272 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.607403 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:39:12.608340 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: NON_VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: true } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.608902 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:12.610323 11007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 LEARNER]: Refusing update from remote peer 97a992b3716d4a66b6ca82dd811968fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 23:39:12.611410 10923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 23:39:12.611711 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.611211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49860)
0810 23:39:12.611364 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.611482 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 23:39:12.615644 11308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:12.615892 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.615480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49866)
0810 23:39:12.615626 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.615722 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 23:39:12.616011 10927 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260810 23:39:12.616168 9848 consensus_peers.cc:487] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd -> Peer d8babcf8bb3647998cf0020b94c67171 (127.30.33.60:39057): Couldn't send request to peer d8babcf8bb3647998cf0020b94c67171. Status: Network error: Client connection negotiation failed: client connection to 127.30.33.60:39057: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:39:12.616283 10927 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.005s
I20260810 23:39:12.616410 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
uuid: "d8babcf8bb3647998cf0020b94c67171"
format_stamp: "Formatted at 2026-08-10 23:39:12 on dist-test-slave-pww0"
server_key: "42a67c7a3f3d4c0c337772d4de882c98"
server_key_iv: "f68f3fab6504d0d8ad5c11c48c243251"
server_key_version: "encryptionkey@0"
I20260810 23:39:12.616802 11311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8029edc52e745609f8f4878a3704ee5 (127.30.33.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.617511 11007 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8029edc52e745609f8f4878a3704ee5 (127.30.33.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.621444 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.621733 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:39:12.621729 10927 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:12.622403 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.622686 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:12.623126 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.623387 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:12.629030 10927 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:39:12.629243 10927 log.cc:1196] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 23:39:12.629799 10927 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Deleting consensus metadata
I20260810 23:39:12.630064 10927 master_runner.cc:315] Copying system tablet from 127.30.33.62:40579
I20260810 23:39:12.635604 10927 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: tablet copy: Beginning tablet copy session from remote peer at address 127.30.33.62:40579
I20260810 23:39:12.660117 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.636534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.30.33.62:40579)
0810 23:39:12.636794 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:12.636802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:12.648222 (+ 11420us) server_negotiation.cc:407] Connection header received
0810 23:39:12.648451 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:12.648457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:12.648551 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:12.648648 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:12.649642 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.650430 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.651844 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.652304 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.652426 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.654373 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:12.654379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:12.654382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:12.654420 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:12.657213 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.657898 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.657905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.657909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.658067 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:12.658524 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:12.658530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:12.658534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:12.658854 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:12.659005 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:12.659448 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 23:39:12.659719 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 23:39:12.660198 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:12.636443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:35800)
0810 23:39:12.648060 (+ 11617us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:12.648082 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:12.648301 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:12.648719 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:12.648725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:12.649213 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:12.649455 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.649467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.650601 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.650607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.651681 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:12.651697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:12.652488 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:12.652495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:12.652685 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:12.653337 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:12.653356 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:12.654143 (+ 787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:12.657356 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.657362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.657367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.657772 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.658206 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:12.658211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:12.658215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:12.658416 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:12.658985 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:12.658990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:12.659180 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 23:39:12.659325 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 23:39:12.659797 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11428,"thread_start_us":11249,"threads_started":1}
I20260810 23:39:12.660980 9908 tablet_copy_service.cc:140] P 97a992b3716d4a66b6ca82dd811968fd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d8babcf8bb3647998cf0020b94c67171 ({username='oryx', principal='oryx/127.30.33.60@KRBTEST.COM'} at 127.0.0.1:35800)
I20260810 23:39:12.661164 9908 tablet_copy_service.cc:161] P 97a992b3716d4a66b6ca82dd811968fd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d8babcf8bb3647998cf0020b94c67171 at {username='oryx', principal='oryx/127.30.33.60@KRBTEST.COM'} at 127.0.0.1:35800: session id = d8babcf8bb3647998cf0020b94c67171-00000000000000000000000000000000
I20260810 23:39:12.663893 9908 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:39:12.665627 10927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:39:12.670660 10927 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: tablet copy: Starting download of 0 data blocks...
I20260810 23:39:12.671041 10927 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: tablet copy: Starting download of 1 WAL segments...
I20260810 23:39:12.673527 10927 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:39:12.677357 9908 tablet_copy_service.cc:342] P 97a992b3716d4a66b6ca82dd811968fd: Request end of tablet copy session d8babcf8bb3647998cf0020b94c67171-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.33.60@KRBTEST.COM'} at 127.0.0.1:35800
I20260810 23:39:12.677513 9908 tablet_copy_service.cc:433] P 97a992b3716d4a66b6ca82dd811968fd: ending tablet copy session d8babcf8bb3647998cf0020b94c67171-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d8babcf8bb3647998cf0020b94c67171
I20260810 23:39:12.688802 10927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:12.691406 11324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:12.697117 11325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:12.704288 11327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:12.704785 10927 server_base.cc:1034] running on GCE node
I20260810 23:39:12.705152 10927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:12.705806 10927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:12.707022 10927 hybrid_clock.cc:648] HybridClock initialized: now 1786405152706958 us; error 39 us; skew 500 ppm
I20260810 23:39:12.707278 10927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:12 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405152, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:12.709468 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 9973
I20260810 23:39:12.709553 10927 init.cc:377] Logged in from keytab as oryx/127.30.33.60@KRBTEST.COM (short username oryx)
I20260810 23:39:12.717034 10927 webserver.cc:460] Webserver started at http://127.30.33.60:45343/ using document root <none> and password file <none>
I20260810 23:39:12.717466 10927 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:12.717579 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:12.720952 10927 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 23:39:12.723119 11342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:12.723620 10927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:39:12.723819 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
uuid: "d8babcf8bb3647998cf0020b94c67171"
format_stamp: "Formatted at 2026-08-10 23:39:12 on dist-test-slave-pww0"
server_key: "42a67c7a3f3d4c0c337772d4de882c98"
server_key_iv: "f68f3fab6504d0d8ad5c11c48c243251"
server_key_version: "encryptionkey@0"
I20260810 23:39:12.724093 10927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:12.736040 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:43999
--local_ip_for_outbound_sockets=127.30.33.1
--tserver_master_addrs=127.30.33.62:40579
--webserver_port=38433
--webserver_interface=127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab
--principal=oryx/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:39:12.745296 10927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:12.749629 10927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:12.792186 10927 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.60:39057
I20260810 23:39:12.792222 11448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.60:39057 every 8 connection(s)
I20260810 23:39:12.793151 10927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
I20260810 23:39:12.798913 11449 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:39:12.802090 11449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Bootstrap starting.
I20260810 23:39:12.816605 11449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:39:12.817127 11449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Bootstrap complete.
I20260810 23:39:12.818094 11449 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } }
I20260810 23:39:12.818338 11449 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8babcf8bb3647998cf0020b94c67171, State: Initialized, Role: LEARNER
I20260810 23:39:12.818596 11449 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } }
I20260810 23:39:12.819516 11457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:12.819788 11449 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:12.819921 11457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:39:12.829028 11477 catalog_manager.cc:1259] Loaded cluster ID: 97c24683486b4c94963bd5fc148581c3
I20260810 23:39:12.829566 11477 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: loading cluster ID for follower catalog manager: success
I20260810 23:39:12.833189 11477 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: acquiring CA information for follower catalog manager: success
I20260810 23:39:12.835847 11477 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:39:13.071156 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.064811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49874)
0810 23:39:13.064997 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:13.065015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:13.065145 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:13.066006 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:13.066012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:13.066063 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:13.066306 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.066317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.067731 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.067736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.068952 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.068969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.070229 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.070235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.070484 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.070531 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:39:13.070634 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:39:13.070824 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:39:13.071183 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.065018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.30.33.60:39057)
0810 23:39:13.065646 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.065650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.065672 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:39:13.065739 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.065743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.065819 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.065937 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:13.066495 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.067552 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.069154 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.070034 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.070164 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.070473 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.070683 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.070831 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":322,"threads_started":1}
I20260810 23:39:13.072347 11406 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 23:39:13.073153 11457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97a992b3716d4a66b6ca82dd811968fd. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:13.073458 11457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: This master's current role is: LEARNER
W20260810 23:39:13.124434 11352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:13.124892 11352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:13.125067 11352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:13.136063 11352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:13.136193 11352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:13.136317 11352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:13.146766 11352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab
--principal=oryx/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:43999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=38433
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:13.148377 11352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:13.149771 11352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:13.164546 11493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:13.164674 11494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:13.165290 11500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:13.165390 11352 server_base.cc:1034] running on GCE node
I20260810 23:39:13.166095 11352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:13.167184 11352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:13.168421 11352 hybrid_clock.cc:648] HybridClock initialized: now 1786405153168330 us; error 84 us; skew 500 ppm
I20260810 23:39:13.168782 11352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:13.172189 11352 init.cc:377] Logged in from keytab as oryx/127.30.33.1@KRBTEST.COM (short username oryx)
I20260810 23:39:13.173820 11352 webserver.cc:460] Webserver started at http://127.30.33.1:38433/ using document root <none> and password file <none>
I20260810 23:39:13.174528 11352 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:13.174655 11352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:13.180727 11352 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:39:13.183957 11510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:13.185546 11352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:39:13.185760 11352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "8b3c2f7a295c4fe891e0044311bc9a99"
format_stamp: "Formatted at 2026-08-10 23:39:09 on dist-test-slave-pww0"
server_key: "512ceff7fce8983f0cef889b103f3896"
server_key_iv: "18d141d917789df712627551cc8ea946"
server_key_version: "encryptionkey@0"
I20260810 23:39:13.186882 11352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:13.215055 11352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:13.220184 11352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:13.220534 11352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:13.221745 11352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:13.223625 11352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:13.223759 11352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:13.223869 11352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:13.223934 11352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:13.256013 11352 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43999
I20260810 23:39:13.256103 11748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43999 every 8 connection(s)
I20260810 23:39:13.257634 11352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:13.267026 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 11352
I20260810 23:39:13.267362 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 10227
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.60@KRBTEST.COM
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.2@KRBTEST.COM
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.61@KRBTEST.COM
I20260810 23:39:13.293334 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.262378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:44359 (local address 127.30.33.60:39057)
0810 23:39:13.262578 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.262584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.264188 (+ 1604us) server_negotiation.cc:407] Connection header received
0810 23:39:13.266059 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.266064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.266149 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.266263 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.269573 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.270769 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.276393 (+ 5624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.277046 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.277190 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.284806 (+ 7616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.284814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.284819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.284865 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.288999 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.290193 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.290200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.290203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.290381 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.291010 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.291015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.291017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.291404 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.291590 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.292730 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.292984 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 23:39:13.293359 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.260765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:36641 (local address 127.30.33.62:40579)
0810 23:39:13.261224 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.261229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.262644 (+ 1415us) server_negotiation.cc:407] Connection header received
0810 23:39:13.265527 (+ 2883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.265535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.265637 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.265758 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.268388 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.269525 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.273660 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.274340 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.274483 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.285069 (+ 10586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.285076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.285080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.285125 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.288587 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.290175 (+ 1588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.290182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.290186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.290361 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.291083 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.291087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.291090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.291519 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.291660 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.292751 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.292999 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260810 23:39:13.294327 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.263545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:34997 (local address 127.30.33.61:41497)
0810 23:39:13.266627 (+ 3082us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.266633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.275762 (+ 9129us) server_negotiation.cc:407] Connection header received
0810 23:39:13.276241 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.276247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.276351 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.276461 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.278799 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.279725 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.281462 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.282125 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.282247 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.286187 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.286196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.286201 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.286248 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.290013 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.290948 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.290954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.290957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.291124 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.291604 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.291609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.291612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.292038 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.292190 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.293716 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.293929 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2946,"thread_start_us":2127,"threads_started":1}
I20260810 23:39:13.294991 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.268603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.30.33.2:41157)
0810 23:39:13.272018 (+ 3415us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.272027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.272598 (+ 571us) server_negotiation.cc:407] Connection header received
0810 23:39:13.272762 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.272772 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.272999 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.273145 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.275081 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.276351 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.277931 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.278527 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.278647 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.286313 (+ 7666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.286336 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.286355 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.286402 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.290236 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.291262 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.291271 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.291274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.291430 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.291929 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.291934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.291937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.292380 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.292689 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.293754 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.294049 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3222,"thread_start_us":3039,"threads_started":1}
I20260810 23:39:13.295318 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.261443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.1:36641)
0810 23:39:13.262454 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:13.262509 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:13.265117 (+ 2608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:13.265979 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:13.265992 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:13.267057 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:13.268129 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.268161 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.271012 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.271018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.273447 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.273466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.274587 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.274595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.274929 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.280234 (+ 5305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:13.280273 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:13.284794 (+ 4521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:13.288801 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.288812 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.288834 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.290006 (+ 1172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.290535 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.290540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.290543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.290941 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.291849 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:13.291853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:13.292260 (+ 407us) client_negotiation.cc:769] Sending connection context
0810 23:39:13.293582 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0810 23:39:13.293890 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":33,"spinlock_wait_cycles":49280,"thread_start_us":850,"threads_started":1}
I20260810 23:39:13.295364 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.263364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.30.33.1:34997)
0810 23:39:13.275646 (+ 12282us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:13.275666 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:13.276079 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:13.277963 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:13.277967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:13.278386 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:13.278636 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.278644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.279903 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.279907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.281255 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.281272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.282499 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.282506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.282698 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.283396 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:13.283421 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:13.285922 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:13.290175 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.290180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.290184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.290802 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.291259 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.291264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.291268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.291417 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.292251 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:13.292255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:13.292385 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 23:39:13.292596 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 23:39:13.293175 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12132,"spinlock_wait_cycles":1792,"thread_start_us":317,"threads_started":1}
I20260810 23:39:13.295431 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.262218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.30.33.1:44359)
0810 23:39:13.264000 (+ 1782us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:13.264021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:13.265122 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:13.266305 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:13.266310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:13.268987 (+ 2677us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:13.269353 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.269367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.273694 (+ 4327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.273699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.276244 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.276255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.277241 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.277246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.277371 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.280227 (+ 2856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:13.280273 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:13.284439 (+ 4166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:13.289181 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.289186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.289189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.290035 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.290536 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.290540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.290543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.290865 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.291690 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:13.291710 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:13.292259 (+ 549us) client_negotiation.cc:769] Sending connection context
0810 23:39:13.293866 (+ 1607us) client_negotiation.cc:240] Negotiation successful
0810 23:39:13.294139 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"mutex_wait_us":28,"spinlock_wait_cycles":59648,"thread_start_us":13567,"threads_started":1}
I20260810 23:39:13.296728 10467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:13.297140 11756 heartbeater.cc:347] Connected to a master server at 127.30.33.61:41497
I20260810 23:39:13.297278 11751 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40579
I20260810 23:39:13.297500 11756 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:13.297508 11751 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:13.298427 11756 heartbeater.cc:511] Master 127.30.33.61:41497 requested a full tablet report, sending...
I20260810 23:39:13.298494 11749 heartbeater.cc:347] Connected to a master server at 127.30.33.60:39057
I20260810 23:39:13.298527 11751 heartbeater.cc:511] Master 127.30.33.62:40579 requested a full tablet report, sending...
I20260810 23:39:13.298617 11749 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:13.299113 11749 heartbeater.cc:511] Master 127.30.33.60:39057 requested a full tablet report, sending...
I20260810 23:39:13.300591 9869 ts_manager.cc:194] Re-registered known tserver with Master: 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:13.301287 10989 ts_manager.cc:194] Registered new tserver with Master: 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:13.301337 11379 ts_manager.cc:194] Registered new tserver with Master: 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:13.301767 9869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:36641
I20260810 23:39:13.398969 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 10227
I20260810 23:39:13.423342 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:41157
--local_ip_for_outbound_sockets=127.30.33.2
--tserver_master_addrs=127.30.33.62:40579
--webserver_port=37665
--webserver_interface=127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab
--principal=oryx/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:39:13.560794 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: NON_VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: true } } }
I20260810 23:39:13.561214 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:13.562889 11773 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: attempting to promote NON_VOTER d8babcf8bb3647998cf0020b94c67171 to VOTER
I20260810 23:39:13.563570 11773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } }
I20260810 23:39:13.565680 11007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Refusing update from remote peer 97a992b3716d4a66b6ca82dd811968fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 23:39:13.565683 11406 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 LEARNER]: Refusing update from remote peer 97a992b3716d4a66b6ca82dd811968fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 23:39:13.566301 11778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 23:39:13.566776 11293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 23:39:13.569242 11293 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8babcf8bb3647998cf0020b94c67171 (127.30.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.572785 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.573172 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:13.573222 11007 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8babcf8bb3647998cf0020b94c67171 (127.30.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.573539 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.574308 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:13.577365 11786 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.577752 11786 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:39:13.577904 11778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.578174 11778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:13.578774 11406 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8babcf8bb3647998cf0020b94c67171 (127.30.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.583556 11787 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97a992b3716d4a66b6ca82dd811968fd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:13.583911 11787 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:39:13.785624 11772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:13.785905 11772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:13.786036 11772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:13.793661 11772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:13.793820 11772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:13.793974 11772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:13.804236 11772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab
--principal=oryx/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:41157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=37665
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:13.805609 11772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:13.806972 11772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:13.820502 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:13.820562 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:13.821024 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:13.822180 11772 server_base.cc:1034] running on GCE node
I20260810 23:39:13.822916 11772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:13.824030 11772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:13.825243 11772 hybrid_clock.cc:648] HybridClock initialized: now 1786405153825192 us; error 49 us; skew 500 ppm
I20260810 23:39:13.825634 11772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:13.829020 11772 init.cc:377] Logged in from keytab as oryx/127.30.33.2@KRBTEST.COM (short username oryx)
I20260810 23:39:13.830652 11772 webserver.cc:460] Webserver started at http://127.30.33.2:37665/ using document root <none> and password file <none>
I20260810 23:39:13.831368 11772 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:13.831527 11772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:13.837671 11772 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 23:39:13.840749 11820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:13.842432 11772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:39:13.842648 11772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "c6a3eb08c9b5401b99a2dc64b2590c53"
format_stamp: "Formatted at 2026-08-10 23:39:10 on dist-test-slave-pww0"
server_key: "0440649ae07245d09ce0533dc1053837"
server_key_iv: "e2840527e822630dd522f249e1a8b130"
server_key_version: "encryptionkey@0"
I20260810 23:39:13.843808 11772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:13.875566 11772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:13.880182 11772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:13.880507 11772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:13.881603 11772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:13.883044 11772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:13.883155 11772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:13.883267 11772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:13.883345 11772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:13.915692 11772 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:41157
I20260810 23:39:13.915772 12019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:41157 every 8 connection(s)
I20260810 23:39:13.917196 11772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:13.922780 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 11772
I20260810 23:39:13.923156 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 10499
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.3@KRBTEST.COM
I20260810 23:39:13.955220 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.924711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.30.33.3:44785)
0810 23:39:13.929384 (+ 4673us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.929396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.930953 (+ 1557us) server_negotiation.cc:407] Connection header received
0810 23:39:13.931055 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.931067 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.931395 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.931614 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.935056 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.937189 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.939373 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.940392 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.940534 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.947101 (+ 6567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.947124 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.947142 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.947200 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.951259 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.952287 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.952297 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.952301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.952498 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.953026 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.953031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.953034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.953516 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.953814 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.954522 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.954804 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4438,"thread_start_us":4264,"threads_started":1}
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:13.956979 10720 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.61@KRBTEST.COM
Aug 10 23:39:13 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405153, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.60@KRBTEST.COM
I20260810 23:39:13.967279 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.920216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:40469 (local address 127.30.33.60:39057)
0810 23:39:13.940084 (+ 19868us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.940090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.942251 (+ 2161us) server_negotiation.cc:407] Connection header received
0810 23:39:13.944582 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.944588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.944685 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.944767 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.948391 (+ 3624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.949578 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.952800 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.953448 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.953527 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.959798 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.959805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.959809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.959856 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.963087 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.964151 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.964159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.964162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.964319 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.964906 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.964911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.964915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.965291 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.965424 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.966547 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.966856 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19726,"thread_start_us":19877,"threads_started":1}
I20260810 23:39:13.967316 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.920351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:45003 (local address 127.30.33.62:40579)
0810 23:39:13.934121 (+ 13770us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.934128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.941208 (+ 7080us) server_negotiation.cc:407] Connection header received
0810 23:39:13.944712 (+ 3504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.944718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.944824 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.944925 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.948073 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.949255 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.952397 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.953029 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.953166 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.958740 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.958748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.958753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.958819 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.963676 (+ 4857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.964607 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.964614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.964617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.964780 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.965239 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.965244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.965247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.965612 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.965745 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.966540 (+ 795us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.966808 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13610,"thread_start_us":13793,"threads_started":1}
I20260810 23:39:13.968551 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.942172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:43061 (local address 127.30.33.61:41497)
0810 23:39:13.942700 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:13.942706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:13.943439 (+ 733us) server_negotiation.cc:407] Connection header received
0810 23:39:13.944576 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:13.944583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:13.944690 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:13.944774 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:13.948069 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.949225 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.952411 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.953038 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.953162 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.959728 (+ 6566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:13.959737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:13.959743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:13.959830 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:13.965301 (+ 5471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.966136 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.966142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.966146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.966311 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:13.966754 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:13.966759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:13.966763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:13.967122 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:13.967269 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:13.967826 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 23:39:13.968060 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":279,"threads_started":1}
I20260810 23:39:13.970211 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.920890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.2:45003)
0810 23:39:13.941002 (+ 20112us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:13.941070 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:13.944197 (+ 3127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:13.945298 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:13.945312 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:13.946528 (+ 1216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:13.947830 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.947863 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.949453 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.949459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.952217 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.952234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.953479 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.953487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.953840 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.954864 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:13.954895 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:13.958430 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:13.963867 (+ 5437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.963872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.963876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.964438 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.964936 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.964940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.964944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.965104 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.965829 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:13.965833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:13.966064 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 23:39:13.967670 (+ 1606us) client_negotiation.cc:240] Negotiation successful
0810 23:39:13.968455 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19678,"mutex_wait_us":343,"spinlock_wait_cycles":55168,"thread_start_us":20489,"threads_started":1}
I20260810 23:39:13.970279 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.942692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.30.33.2:43061)
0810 23:39:13.943289 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:13.943311 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:13.944210 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:13.945298 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:13.945312 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:13.946595 (+ 1283us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:13.947830 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.947862 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.949453 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.949459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.952224 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.952234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.954295 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.954303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.954479 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.956230 (+ 1751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:13.956248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:13.959402 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:13.965459 (+ 6057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.965464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.965467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.965962 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.966456 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.966461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.966464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.966626 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.967358 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:13.967362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:13.967537 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 23:39:13.968751 (+ 1214us) client_negotiation.cc:240] Negotiation successful
0810 23:39:13.969141 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":654,"spinlock_wait_cycles":92160,"thread_start_us":293,"threads_started":1}
I20260810 23:39:13.970309 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:13.920890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.30.33.2:40469)
0810 23:39:13.941851 (+ 20961us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:13.941871 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:13.944197 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:13.945298 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:13.945312 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:13.946528 (+ 1216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:13.947844 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.947862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.949853 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.949858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.952633 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:13.952650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:13.953766 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:13.953771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:13.953896 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:13.955580 (+ 1684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:13.955604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:13.959520 (+ 3916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:13.963261 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.963273 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.963293 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.963982 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.964468 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:13.964473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:13.964477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:13.964772 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:13.965536 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:13.965544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:13.966065 (+ 521us) client_negotiation.cc:769] Sending connection context
0810 23:39:13.967964 (+ 1899us) client_negotiation.cc:240] Negotiation successful
0810 23:39:13.968978 (+ 1014us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20788,"mutex_wait_us":1037,"spinlock_wait_cycles":85120,"thread_start_us":19385,"threads_started":1}
I20260810 23:39:13.972461 12021 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40579
I20260810 23:39:13.972822 12021 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:13.973660 12021 heartbeater.cc:511] Master 127.30.33.62:40579 requested a full tablet report, sending...
I20260810 23:39:13.974411 12022 heartbeater.cc:347] Connected to a master server at 127.30.33.61:41497
I20260810 23:39:13.974526 12022 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:13.974979 12022 heartbeater.cc:511] Master 127.30.33.61:41497 requested a full tablet report, sending...
I20260810 23:39:13.975255 12020 heartbeater.cc:347] Connected to a master server at 127.30.33.60:39057
I20260810 23:39:13.975337 12020 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:13.975449 9869 ts_manager.cc:194] Re-registered known tserver with Master: c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157)
I20260810 23:39:13.975756 12020 heartbeater.cc:511] Master 127.30.33.60:39057 requested a full tablet report, sending...
I20260810 23:39:13.976624 9869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:45003
I20260810 23:39:13.976651 10989 ts_manager.cc:194] Registered new tserver with Master: c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157)
I20260810 23:39:13.977484 11379 ts_manager.cc:194] Registered new tserver with Master: c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157)
I20260810 23:39:14.047581 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 10499
I20260810 23:39:14.074836 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:44785
--local_ip_for_outbound_sockets=127.30.33.3
--tserver_master_addrs=127.30.33.62:40579
--webserver_port=36827
--webserver_interface=127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab
--principal=oryx/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:39:14.304842 11751 heartbeater.cc:503] Master 127.30.33.62:40579 was elected leader, sending a full tablet report...
W20260810 23:39:14.440948 12050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:14.441300 12050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:14.441439 12050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:14.449112 12050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:14.449258 12050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:14.449420 12050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:14.459865 12050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab
--principal=oryx/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:44785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=36827
--tserver_master_addrs=127.30.33.62:40579,127.30.33.61:41497,127.30.33.60:39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:14.461426 12050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:14.462911 12050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:14.477460 12058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:14.477396 12056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:14.477428 12061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:14.478339 12050 server_base.cc:1034] running on GCE node
I20260810 23:39:14.479030 12050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:14.480163 12050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:14.481407 12050 hybrid_clock.cc:648] HybridClock initialized: now 1786405154481333 us; error 57 us; skew 500 ppm
I20260810 23:39:14.481747 12050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:14 dist-test-slave-pww0 krb5kdc[9794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405154, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:14.485086 12050 init.cc:377] Logged in from keytab as oryx/127.30.33.3@KRBTEST.COM (short username oryx)
I20260810 23:39:14.486754 12050 webserver.cc:460] Webserver started at http://127.30.33.3:36827/ using document root <none> and password file <none>
I20260810 23:39:14.487558 12050 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:14.487711 12050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:14.494338 12050 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:39:14.497553 12076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:14.499224 12050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:39:14.499434 12050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "8a41d48a84aa4d34a000ba1a59f213c3"
format_stamp: "Formatted at 2026-08-10 23:39:11 on dist-test-slave-pww0"
server_key: "e14c73c2fc1f4ec5bb928d62d34dcdf0"
server_key_iv: "b75a11a97a8bd3b3d76c95cdf836425f"
server_key_version: "encryptionkey@0"
I20260810 23:39:14.500573 12050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:14.526898 12050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:14.531980 12050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:14.532305 12050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:14.533376 12050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:14.534938 12050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:14.535041 12050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:14.535167 12050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:14.535240 12050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:14.567266 12050 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:44785
I20260810 23:39:14.567337 12310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:44785 every 8 connection(s)
I20260810 23:39:14.568847 12050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:14.574739 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 12050
Aug 10 23:39:14 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405154, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
Aug 10 23:39:14 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405154, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.60@KRBTEST.COM
I20260810 23:39:14.602753 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.572613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.3:58185)
0810 23:39:14.573580 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.573645 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.576439 (+ 2794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.580224 (+ 3785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.580228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.580649 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:14.581376 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.581404 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.582931 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.582937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.585594 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.585610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.586576 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.586583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.586813 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.587780 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:14.587806 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:14.592491 (+ 4685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:14.597310 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:14.597324 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:14.597347 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:14.598032 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:14.598530 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:14.598535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:14.598539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:14.598821 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:14.599645 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:14.599653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:14.600158 (+ 505us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.600522 (+ 364us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.601118 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":93,"spinlock_wait_cycles":2816,"thread_start_us":360,"threads_started":1}
I20260810 23:39:14.604064 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.572064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:58185 (local address 127.30.33.62:40579)
0810 23:39:14.572547 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.572553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.573799 (+ 1246us) server_negotiation.cc:407] Connection header received
0810 23:39:14.579961 (+ 6162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.579968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.580056 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.581010 (+ 954us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:14.581587 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.582701 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.585748 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.586383 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.586507 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.592940 (+ 6433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:14.592948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:14.592954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:14.593005 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:14.597087 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.598198 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.598205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.598209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.598382 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.598965 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.598970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.598973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.599380 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:14.599558 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.603351 (+ 3793us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.603673 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260810 23:39:14.605609 12312 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40579
I20260810 23:39:14.606011 12312 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:14.607059 12312 heartbeater.cc:511] Master 127.30.33.62:40579 requested a full tablet report, sending...
I20260810 23:39:14.607239 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.572360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:56635 (local address 127.30.33.60:39057)
0810 23:39:14.572784 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.572790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.577638 (+ 4848us) server_negotiation.cc:407] Connection header received
0810 23:39:14.577904 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.577911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.578016 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.578102 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:14.582085 (+ 3983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.583178 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.585457 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.586059 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.586183 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.597745 (+ 11562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:14.597753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:14.597758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:14.597805 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:14.601427 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.602991 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.602997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.603001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.603162 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.604005 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.604010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.604013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.604440 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:14.604578 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.606595 (+ 2017us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.606854 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
Aug 10 23:39:14 dist-test-slave-pww0 krb5kdc[9794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405154, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.61@KRBTEST.COM
I20260810 23:39:14.608434 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.572927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.30.33.3:56635)
0810 23:39:14.577517 (+ 4590us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.577539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.577756 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.578634 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.578648 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.580458 (+ 1810us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:14.581892 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.581904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.583374 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.583379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.585286 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.585302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.586504 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.586512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.586809 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.587686 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:14.587727 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:14.596601 (+ 8874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:14.601637 (+ 5036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:14.601642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:14.601646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:14.602170 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:14.603342 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:14.603346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:14.603349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:14.603544 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:14.604681 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:14.604686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:14.604857 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.607622 (+ 2765us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.607937 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4448,"mutex_wait_us":350,"spinlock_wait_cycles":15232,"thread_start_us":6682,"threads_started":1}
I20260810 23:39:14.609757 9869 ts_manager.cc:194] Re-registered known tserver with Master: 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785)
I20260810 23:39:14.611006 9869 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:58185
I20260810 23:39:14.613070 12311 heartbeater.cc:347] Connected to a master server at 127.30.33.60:39057
I20260810 23:39:14.613219 12311 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:14.613683 12311 heartbeater.cc:511] Master 127.30.33.60:39057 requested a full tablet report, sending...
I20260810 23:39:14.615559 11379 ts_manager.cc:194] Registered new tserver with Master: 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785)
I20260810 23:39:14.617609 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.575883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.30.33.3:49655)
0810 23:39:14.579883 (+ 4000us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.579901 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.580008 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.583439 (+ 3431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.583444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.584167 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:14.586051 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.586064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.588276 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.588282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.591445 (+ 3163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.591493 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.604857 (+ 13364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.604864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.605105 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.605766 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:14.605788 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:14.609061 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:14.613783 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:14.613788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:14.613792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:14.614309 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:14.615290 (+ 981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:14.615297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:14.615301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:14.615554 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:14.616547 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:14.616552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:14.616731 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.616870 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.617193 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3866,"thread_start_us":1310,"threads_started":1}
I20260810 23:39:14.618665 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.576262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:49655 (local address 127.30.33.61:41497)
0810 23:39:14.582932 (+ 6670us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.582939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.582961 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:39:14.583028 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.583034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.583117 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.583237 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:14.586245 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.588098 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.591703 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.592422 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.592533 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.609859 (+ 17326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:14.609867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:14.609872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:14.609920 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:14.613555 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.614468 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.614474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.614478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.614647 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.615687 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.615692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.615695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.616057 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:14.616242 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.618017 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.618276 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6518,"thread_start_us":7583,"threads_started":1}
I20260810 23:39:14.627068 12315 heartbeater.cc:347] Connected to a master server at 127.30.33.61:41497
I20260810 23:39:14.627269 12315 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:14.627516 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.590335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.30.33.62:40579)
0810 23:39:14.604672 (+ 14337us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.604677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.605994 (+ 1317us) server_negotiation.cc:407] Connection header received
0810 23:39:14.606284 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.606290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.606383 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.606454 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:14.609746 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.611017 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.613270 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.613953 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.614054 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.619122 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:14.619130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:14.619136 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:14.619185 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:14.623384 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.624647 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.624655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.624659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.624910 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.625473 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.625479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.625483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.625938 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:14.626187 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.626834 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.627102 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14229,"thread_start_us":15100,"threads_started":1}
I20260810 23:39:14.627851 12315 heartbeater.cc:511] Master 127.30.33.61:41497 requested a full tablet report, sending...
I20260810 23:39:14.629248 10989 ts_manager.cc:194] Registered new tserver with Master: 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785)
I20260810 23:39:14.640673 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.590462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.30.33.60:39057)
0810 23:39:14.607569 (+ 17107us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.607581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.624029 (+ 16448us) server_negotiation.cc:407] Connection header received
0810 23:39:14.624404 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.624411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.624566 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.624643 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:14.627923 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.628962 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.630893 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.631439 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.631607 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.634482 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:14.634489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:14.634493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:14.634530 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:14.637295 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.637981 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.637987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.637991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.638175 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.638682 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.638689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.638693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.639043 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:14.639340 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.639756 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.640280 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17005,"thread_start_us":32615,"threads_started":1}
I20260810 23:39:14.634078 9869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:39:14.644088 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.608135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.30.33.61:41497)
0810 23:39:14.618883 (+ 10748us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.618889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.627120 (+ 8231us) server_negotiation.cc:407] Connection header received
0810 23:39:14.627542 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.627548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.627915 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.628049 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:14.630062 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.631089 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.633091 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.633725 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.633856 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.637259 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:14.637267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:14.637272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:14.637319 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:14.640830 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.641752 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.641757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.641760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.641894 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:14.642418 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:14.642423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:14.642426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:14.642833 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:14.643101 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.643430 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.643720 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10647,"thread_start_us":17952,"threads_started":1}
W20260810 23:39:14.644533 9869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:14.689649 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.668298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41157 (local address 127.0.0.1:59788)
0810 23:39:14.681328 (+ 13030us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.681347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.681489 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.682507 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.682512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.682539 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:14.682884 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.682898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.684819 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.684825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.686909 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.686929 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.688762 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.688769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.688946 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.688983 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.689088 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.689260 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12873,"thread_start_us":12725,"threads_started":1}
I20260810 23:39:14.690310 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.669307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.30.33.1:43999)
0810 23:39:14.680103 (+ 10796us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.680114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.682380 (+ 2266us) server_negotiation.cc:407] Connection header received
0810 23:39:14.682531 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.682540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.682816 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.683033 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:14.683598 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.684618 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.687351 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.688832 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.688958 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.689170 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.689519 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.689699 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10608,"thread_start_us":10478,"threads_started":1}
I20260810 23:39:14.690294 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.668874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.30.33.2:41157)
0810 23:39:14.680842 (+ 11968us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.680854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.681469 (+ 615us) server_negotiation.cc:407] Connection header received
0810 23:39:14.681674 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.681683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.682009 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.682248 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:14.683085 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.684611 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.687087 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.688557 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.688698 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.688940 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.689793 (+ 853us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.689972 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11510,"thread_start_us":11404,"threads_started":1}
I20260810 23:39:14.690742 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.668225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.30.33.3:44785)
0810 23:39:14.682967 (+ 14742us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.682976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.683010 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:39:14.683112 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.683120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.683353 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.684191 (+ 838us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:14.684485 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.686142 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.688317 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.689703 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.689839 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.690052 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.690267 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.690429 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14561,"thread_start_us":14386,"threads_started":1}
I20260810 23:39:14.690732 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.667862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:44785 (local address 127.0.0.1:50770)
0810 23:39:14.669465 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.669488 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.669636 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.683604 (+ 13968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.683610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.683636 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:14.683992 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.684003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.686295 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.686301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.688138 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.688151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.689888 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.689895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.690069 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.690102 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.690189 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.690349 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"thread_start_us":14090,"threads_started":1}
I20260810 23:39:14.690768 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.668753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43999 (local address 127.0.0.1:40508)
0810 23:39:14.682213 (+ 13460us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.682241 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.682371 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.683027 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.683031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.683057 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:14.683401 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.683413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.684826 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.684832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.687172 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.687188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.689086 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.689092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.689289 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.689313 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.689427 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.689567 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13333,"thread_start_us":309,"threads_started":1}
I20260810 23:39:14.695101 11912 tablet_service.cc:1467] Processing CreateTablet for tablet 519d141d8a4442e490213ee177feceb7 (DEFAULT_TABLE table=test-table [id=fab8ff3b11c244aa908ab1cefc758479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:14.695218 12176 tablet_service.cc:1467] Processing CreateTablet for tablet 519d141d8a4442e490213ee177feceb7 (DEFAULT_TABLE table=test-table [id=fab8ff3b11c244aa908ab1cefc758479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:14.695209 11608 tablet_service.cc:1467] Processing CreateTablet for tablet 519d141d8a4442e490213ee177feceb7 (DEFAULT_TABLE table=test-table [id=fab8ff3b11c244aa908ab1cefc758479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:14.696473 11912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519d141d8a4442e490213ee177feceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:14.696638 12176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519d141d8a4442e490213ee177feceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:14.696643 11608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519d141d8a4442e490213ee177feceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:14.709836 12376 tablet_bootstrap.cc:492] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Bootstrap starting.
I20260810 23:39:14.713006 12377 tablet_bootstrap.cc:492] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Bootstrap starting.
I20260810 23:39:14.713754 12376 tablet_bootstrap.cc:654] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:14.715045 12379 tablet_bootstrap.cc:492] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Bootstrap starting.
I20260810 23:39:14.715546 12376 log.cc:824] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:14.717607 12377 tablet_bootstrap.cc:654] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:14.718297 12379 tablet_bootstrap.cc:654] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:14.719084 12377 log.cc:824] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:14.719635 12379 log.cc:824] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:14.725277 12376 tablet_bootstrap.cc:492] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: No bootstrap required, opened a new log
I20260810 23:39:14.725395 12377 tablet_bootstrap.cc:492] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: No bootstrap required, opened a new log
I20260810 23:39:14.725621 12376 ts_tablet_manager.cc:1397] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260810 23:39:14.725701 12377 ts_tablet_manager.cc:1397] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260810 23:39:14.725750 12379 tablet_bootstrap.cc:492] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: No bootstrap required, opened a new log
I20260810 23:39:14.726001 12379 ts_tablet_manager.cc:1397] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260810 23:39:14.733754 12376 raft_consensus.cc:348] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.734202 12376 raft_consensus.cc:374] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:14.734331 12376 raft_consensus.cc:729] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a3eb08c9b5401b99a2dc64b2590c53, State: Initialized, Role: FOLLOWER
I20260810 23:39:14.735021 12376 consensus_queue.cc:260] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.736446 12377 raft_consensus.cc:348] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.736929 12377 raft_consensus.cc:374] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:14.737053 12377 raft_consensus.cc:729] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a41d48a84aa4d34a000ba1a59f213c3, State: Initialized, Role: FOLLOWER
I20260810 23:39:14.737486 12021 heartbeater.cc:503] Master 127.30.33.62:40579 was elected leader, sending a full tablet report...
I20260810 23:39:14.737406 12379 raft_consensus.cc:348] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.737861 12379 raft_consensus.cc:374] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:14.737954 12379 raft_consensus.cc:729] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3c2f7a295c4fe891e0044311bc9a99, State: Initialized, Role: FOLLOWER
I20260810 23:39:14.737860 12377 consensus_queue.cc:260] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.738605 12379 consensus_queue.cc:260] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.739375 12376 ts_tablet_manager.cc:1428] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260810 23:39:14.747170 12379 ts_tablet_manager.cc:1428] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Time spent starting tablet: real 0.021s user 0.013s sys 0.008s
I20260810 23:39:14.747645 12312 heartbeater.cc:503] Master 127.30.33.62:40579 was elected leader, sending a full tablet report...
I20260810 23:39:14.749202 12377 ts_tablet_manager.cc:1428] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Time spent starting tablet: real 0.023s user 0.017s sys 0.005s
W20260810 23:39:14.761147 11760 tablet.cc:2369] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:39:14.822065 12316 tablet.cc:2369] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:39:14.895012 12391 raft_consensus.cc:482] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:14.895380 12391 raft_consensus.cc:504] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.897258 12391 leader_election.cc:288] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157), 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:14.905100 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.897865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:40663 (local address 127.30.33.1:43999)
0810 23:39:14.898095 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.898099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.898116 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:39:14.898180 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.898185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.898252 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.898353 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:14.898920 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.900074 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.902900 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.904054 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.904203 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.904321 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.904664 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.904812 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:39:14.905138 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.897766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43999 (local address 127.30.33.3:40663)
0810 23:39:14.897979 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.897994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.898067 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.898377 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.898382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.898422 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:14.898754 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.898769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.900237 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.900241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.902682 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.902702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.904221 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.904228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.904478 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.904512 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.904602 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.904821 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":226,"spinlock_wait_cycles":11136}
I20260810 23:39:14.905613 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.897494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41157 (local address 127.30.33.3:46757)
0810 23:39:14.897838 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:14.897862 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:14.898009 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:14.898377 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:14.898382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:14.898425 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:14.898754 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.898769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.900229 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.900234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.902682 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:14.902702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.904528 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.904535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:14.904768 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.904801 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:39:14.904973 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 23:39:14.905109 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":247,"spinlock_wait_cycles":9472}
I20260810 23:39:14.905682 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.897631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:46757 (local address 127.30.33.2:41157)
0810 23:39:14.897844 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.897850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.898038 (+ 188us) server_negotiation.cc:407] Connection header received
0810 23:39:14.898145 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.898151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.898248 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.898353 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:14.898920 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.900079 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.902908 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.904144 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.904297 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.904406 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.905296 (+ 890us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.905438 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:39:14.906949 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d141d8a4442e490213ee177feceb7" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" is_pre_election: true
I20260810 23:39:14.906942 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d141d8a4442e490213ee177feceb7" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" is_pre_election: true
I20260810 23:39:14.907441 11654 raft_consensus.cc:2454] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in term 0.
I20260810 23:39:14.907435 11947 raft_consensus.cc:2454] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in term 0.
I20260810 23:39:14.908209 12079 leader_election.cc:302] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a41d48a84aa4d34a000ba1a59f213c3, 8b3c2f7a295c4fe891e0044311bc9a99, c6a3eb08c9b5401b99a2dc64b2590c53; no voters:
I20260810 23:39:14.908645 12391 raft_consensus.cc:2781] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:14.908814 12391 raft_consensus.cc:482] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:14.908927 12391 raft_consensus.cc:3037] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:14.911533 12391 raft_consensus.cc:504] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:14.912207 12391 leader_election.cc:288] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 election: Requested vote from peers c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157), 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:14.912734 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d141d8a4442e490213ee177feceb7" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53"
I20260810 23:39:14.912820 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d141d8a4442e490213ee177feceb7" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3c2f7a295c4fe891e0044311bc9a99"
I20260810 23:39:14.913002 11947 raft_consensus.cc:3037] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:14.913039 11654 raft_consensus.cc:3037] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:14.916407 11654 raft_consensus.cc:2454] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in term 1.
I20260810 23:39:14.916412 11947 raft_consensus.cc:2454] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in term 1.
I20260810 23:39:14.916971 12079 leader_election.cc:302] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a41d48a84aa4d34a000ba1a59f213c3, c6a3eb08c9b5401b99a2dc64b2590c53; no voters:
I20260810 23:39:14.917320 12391 raft_consensus.cc:2781] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:14.917830 12391 raft_consensus.cc:686] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 LEADER]: Becoming Leader. State: Replica: 8a41d48a84aa4d34a000ba1a59f213c3, State: Running, Role: LEADER
I20260810 23:39:14.918368 12391 consensus_queue.cc:237] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
W20260810 23:39:14.920809 12023 tablet.cc:2369] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:39:14.924151 9869 catalog_manager.cc:5495] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3). New cstate: current_term: 1 leader_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20260810 23:39:14.981042 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:14.975587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.30.33.3:44785)
0810 23:39:14.975866 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:14.975872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:14.975895 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:39:14.976017 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:14.976021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:14.976117 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:14.976262 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:14.976671 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.977402 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.978751 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:14.979164 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:14.979280 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:14.979878 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:14.980416 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:14.980564 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:14.980680 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:39:14.980781 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:39:14.990562 11654 raft_consensus.cc:1261] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Refusing update from remote peer 8a41d48a84aa4d34a000ba1a59f213c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:14.990561 11947 raft_consensus.cc:1261] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Refusing update from remote peer 8a41d48a84aa4d34a000ba1a59f213c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:14.991878 12397 consensus_queue.cc:1035] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:14.992363 12391 consensus_queue.cc:1035] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:15.005631 12401 mvcc.cc:205] Tried to move back new op lower bound from 7317115514828627968 to 7317115514549436416. Current Snapshot: MvccSnapshot[applied={T|T < 7317115514828627968}]
I20260810 23:39:15.037300 9869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:15.037612 9869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:15.045770 9869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97a992b3716d4a66b6ca82dd811968fd: Sending DeleteTablet for 3 replicas of tablet 519d141d8a4442e490213ee177feceb7
I20260810 23:39:15.046980 12176 tablet_service.cc:1514] Processing DeleteTablet for tablet 519d141d8a4442e490213ee177feceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:15 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:50770
I20260810 23:39:15.047740 11608 tablet_service.cc:1514] Processing DeleteTablet for tablet 519d141d8a4442e490213ee177feceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:15 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:40508
I20260810 23:39:15.047772 11912 tablet_service.cc:1514] Processing DeleteTablet for tablet 519d141d8a4442e490213ee177feceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:15 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:59788
I20260810 23:39:15.059890 12418 tablet_replica.cc:330] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: stopping tablet replica
I20260810 23:39:15.060685 12418 raft_consensus.cc:2229] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:15.061949 12417 tablet_replica.cc:330] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: stopping tablet replica
I20260810 23:39:15.062400 12417 raft_consensus.cc:2229] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:15.062870 12417 raft_consensus.cc:2258] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:15.063090 12416 tablet_replica.cc:330] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: stopping tablet replica
I20260810 23:39:15.061337 12418 raft_consensus.cc:2258] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:15.063640 12416 raft_consensus.cc:2229] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:39:15.064425 12416 raft_consensus.cc:2258] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:15.065605 12418 ts_tablet_manager.cc:1905] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:15.066107 12417 ts_tablet_manager.cc:1905] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:15.066577 12416 ts_tablet_manager.cc:1905] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:15.073724 12417 ts_tablet_manager.cc:1918] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:15.073730 12418 ts_tablet_manager.cc:1918] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:15.073949 12417 log.cc:1196] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/519d141d8a4442e490213ee177feceb7
I20260810 23:39:15.074174 12418 log.cc:1196] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/519d141d8a4442e490213ee177feceb7
I20260810 23:39:15.074837 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 9814
I20260810 23:39:15.074922 12416 ts_tablet_manager.cc:1918] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:15.075119 12416 log.cc:1196] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/519d141d8a4442e490213ee177feceb7
I20260810 23:39:15.075851 12418 ts_tablet_manager.cc:1939] T 519d141d8a4442e490213ee177feceb7 P 8b3c2f7a295c4fe891e0044311bc9a99: Deleting consensus metadata
I20260810 23:39:15.075812 12416 ts_tablet_manager.cc:1939] T 519d141d8a4442e490213ee177feceb7 P 8a41d48a84aa4d34a000ba1a59f213c3: Deleting consensus metadata
I20260810 23:39:15.076200 9924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:15.079339 9848 catalog_manager.cc:4841] TS 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785): tablet 519d141d8a4442e490213ee177feceb7 (table test-table [id=fab8ff3b11c244aa908ab1cefc758479]) successfully deleted
I20260810 23:39:15.082676 12417 ts_tablet_manager.cc:1939] T 519d141d8a4442e490213ee177feceb7 P c6a3eb08c9b5401b99a2dc64b2590c53: Deleting consensus metadata
I20260810 23:39:15.102007 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:15.050942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.30.33.61:41497)
0810 23:39:15.051122 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:15.051129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:15.074609 (+ 23480us) server_negotiation.cc:407] Connection header received
0810 23:39:15.074851 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:15.074857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:15.074954 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:15.075123 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:15.077657 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.078936 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.081020 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.081748 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.081903 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:15.085677 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:15.085686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:15.085692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:15.085762 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:15.089680 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.090878 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.090884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.090886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.091105 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.100186 (+ 9081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.100192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.100194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.100582 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:15.100882 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:15.101354 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:39:15.101608 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:39:15.115899 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:15.073312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.30.33.60:39057)
0810 23:39:15.073535 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:15.073542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:15.102539 (+ 28997us) server_negotiation.cc:407] Connection header received
0810 23:39:15.102842 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:15.102848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:15.102940 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:15.103013 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:15.104842 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.105744 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.107067 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.107549 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.107692 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:15.110285 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:15.110291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:15.110294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:15.110334 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:15.113099 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.113816 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.113823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.113826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.113966 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.114360 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.114366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.114369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.114687 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:15.114942 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:15.115277 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:39:15.115534 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:39:15.260444 9843 catalog_manager.cc:4841] TS 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999): tablet 519d141d8a4442e490213ee177feceb7 (table test-table [id=fab8ff3b11c244aa908ab1cefc758479]) successfully deleted
W20260810 23:39:15.260776 9843 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.30.33.1:43999, user_credentials={real_user=slave}} blocked reactor thread for 180303us
I20260810 23:39:15.261258 9845 catalog_manager.cc:4841] TS c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157): tablet 519d141d8a4442e490213ee177feceb7 (table test-table [id=fab8ff3b11c244aa908ab1cefc758479]) successfully deleted
W20260810 23:39:15.261446 9845 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.30.33.2:41157, user_credentials={real_user=slave}} blocked reactor thread for 176822us
I20260810 23:39:15.261968 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 9814
I20260810 23:39:15.328434 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:15.306811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.30.33.60:39057)
0810 23:39:15.307059 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:15.307066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:15.308093 (+ 1027us) server_negotiation.cc:407] Connection header received
0810 23:39:15.308628 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:15.308635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:15.308983 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:15.309106 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:15.311749 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.312992 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.315381 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.316158 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.316272 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:15.320095 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:15.320100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:15.320104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:15.320149 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:15.323943 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.325836 (+ 1893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.325842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.325845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.326074 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.326538 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.326544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.326547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.326946 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:15.327217 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:15.327626 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 23:39:15.327881 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:39:15.331923 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:15.308036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.30.33.61:41497)
0810 23:39:15.308297 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:15.308304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:15.313718 (+ 5414us) server_negotiation.cc:407] Connection header received
0810 23:39:15.314341 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:15.314347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:15.314437 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:15.314512 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:15.317103 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.319158 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.321156 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:15.321859 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:15.321990 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:15.325011 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:15.325019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:15.325024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:15.325074 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:15.328389 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.329299 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.329304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.329306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.329438 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:15.330167 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:15.330173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:15.330177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:15.330634 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:15.330906 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:15.331280 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 23:39:15.331580 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 23:39:16.026486 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.025619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.2:33325)
0810 23:39:16.026132 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.026262 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":263,"threads_started":1}
W20260810 23:39:16.026988 12021 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:40579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.029825 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.029012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.1:51047)
0810 23:39:16.029498 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.029652 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":219,"threads_started":1}
W20260810 23:39:16.030210 11751 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:40579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.033298 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.032438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.3:39067)
0810 23:39:16.032935 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.033092 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":272,"threads_started":1}
W20260810 23:39:16.033823 12312 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:40579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.545722 12437 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97a992b3716d4a66b6ca82dd811968fd)
I20260810 23:39:16.545955 12437 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } }
I20260810 23:39:16.546028 12439 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 97a992b3716d4a66b6ca82dd811968fd)
I20260810 23:39:16.546295 12439 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } }
I20260810 23:39:16.546850 12437 leader_election.cc:288] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579), c8029edc52e745609f8f4878a3704ee5 (127.30.33.61:41497)
I20260810 23:39:16.548062 12439 leader_election.cc:288] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579), d8babcf8bb3647998cf0020b94c67171 (127.30.33.60:39057)
I20260810 23:39:16.548426 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.547123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36180)
0810 23:39:16.547696 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.547919 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":249,"threads_started":1}
W20260810 23:39:16.549749 11346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.558199 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.547914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:41497 (local address 127.0.0.1:54772)
0810 23:39:16.548753 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.548774 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:16.548992 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:16.550382 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:16.550388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:16.550424 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:16.550710 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.550724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.552744 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.552752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.555060 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.555087 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.557132 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.557141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.557415 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.557468 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 23:39:16.557561 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:39:16.557757 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":1377,"threads_started":1}
I20260810 23:39:16.559252 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.548489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36184)
0810 23:39:16.558641 (+ 10152us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.558917 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10028,"thread_start_us":9859,"threads_started":1}
I20260810 23:39:16.560031 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.548015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.30.33.61:41497)
0810 23:39:16.549888 (+ 1873us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:16.549894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:16.549913 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:16.550061 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:16.550067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:16.550168 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:16.550871 (+ 703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:16.551112 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.552508 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.555396 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.556861 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.558785 (+ 1924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.558938 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:16.559014 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 23:39:16.559186 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735,"thread_start_us":1545,"threads_started":1}
W20260810 23:39:16.559698 10965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.561340 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.560853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36196)
0810 23:39:16.561036 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.561141 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 23:39:16.561586 11007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8babcf8bb3647998cf0020b94c67171" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8029edc52e745609f8f4878a3704ee5" is_pre_election: true
W20260810 23:39:16.561600 11346 leader_election.cc:334] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579): Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.561920 11007 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8babcf8bb3647998cf0020b94c67171 in term 1.
I20260810 23:39:16.562726 11346 leader_election.cc:302] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8029edc52e745609f8f4878a3704ee5, d8babcf8bb3647998cf0020b94c67171; no voters: 97a992b3716d4a66b6ca82dd811968fd
I20260810 23:39:16.563083 12437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:39:16.563221 12437 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97a992b3716d4a66b6ca82dd811968fd)
I20260810 23:39:16.563325 12437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:39:16.564204 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.563598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36204)
0810 23:39:16.563790 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.563977 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:39:16.564456 10965 leader_election.cc:334] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579): Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.565788 12437 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } }
I20260810 23:39:16.566557 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.549929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:39057 (local address 127.0.0.1:49908)
0810 23:39:16.559538 (+ 9609us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.559572 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:16.559720 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:16.560672 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:16.560678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:16.560704 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:16.560955 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.560968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.562359 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.562365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.564517 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.564538 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.565795 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.565801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.565990 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.566027 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:39:16.566117 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:39:16.566268 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9497,"thread_start_us":10467,"threads_started":1}
I20260810 23:39:16.566847 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.558874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.30.33.60:39057)
0810 23:39:16.560152 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:16.560159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:16.560206 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:39:16.560279 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:16.560285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:16.560365 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:16.560468 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:16.561142 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.562221 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.564753 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.565609 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.565727 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.565848 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:16.566168 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 23:39:16.566350 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134,"thread_start_us":1389,"threads_started":1}
I20260810 23:39:16.566570 12437 leader_election.cc:288] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 2 election: Requested vote from peers 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579), c8029edc52e745609f8f4878a3704ee5 (127.30.33.61:41497)
I20260810 23:39:16.567960 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.567577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36210)
0810 23:39:16.567724 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.567813 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:39:16.568259 11007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8babcf8bb3647998cf0020b94c67171" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8029edc52e745609f8f4878a3704ee5"
I20260810 23:39:16.568348 11406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8029edc52e745609f8f4878a3704ee5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d8babcf8bb3647998cf0020b94c67171" is_pre_election: true
I20260810 23:39:16.568495 11007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:39:16.568732 11406 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8029edc52e745609f8f4878a3704ee5 in current term 2: Already voted for candidate d8babcf8bb3647998cf0020b94c67171 in this term.
I20260810 23:39:16.569356 10967 leader_election.cc:302] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8029edc52e745609f8f4878a3704ee5; no voters: 97a992b3716d4a66b6ca82dd811968fd, d8babcf8bb3647998cf0020b94c67171
I20260810 23:39:16.571550 11007 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8babcf8bb3647998cf0020b94c67171 in term 2.
I20260810 23:39:16.571594 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.571179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36226)
0810 23:39:16.571351 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.571424 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:39:16.571727 12439 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 23:39:16.571796 11346 leader_election.cc:334] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579): Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
I20260810 23:39:16.572099 11346 leader_election.cc:302] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8029edc52e745609f8f4878a3704ee5, d8babcf8bb3647998cf0020b94c67171; no voters: 97a992b3716d4a66b6ca82dd811968fd
I20260810 23:39:16.572396 12437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:39:16.572849 12437 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [term 2 LEADER]: Becoming Leader. State: Replica: d8babcf8bb3647998cf0020b94c67171, State: Running, Role: LEADER
I20260810 23:39:16.573231 12437 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } }
I20260810 23:39:16.576126 12459 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8babcf8bb3647998cf0020b94c67171. Latest consensus state: current_term: 2 leader_uuid: "d8babcf8bb3647998cf0020b94c67171" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:16.576519 12459 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:16.577157 12461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:16.581264 12461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:16.582139 12461 catalog_manager.cc:1259] Loaded cluster ID: 97c24683486b4c94963bd5fc148581c3
I20260810 23:39:16.582222 12461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:16.583367 12461 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:16.584213 12461 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Loaded TSK: 0
I20260810 23:39:16.585203 12461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:39:16.632371 11379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:39:16.639870 11379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:16.644855 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.644387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36252)
0810 23:39:16.644534 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.644637 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 23:39:16.645310 11007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [term 2 FOLLOWER]: Refusing update from remote peer d8babcf8bb3647998cf0020b94c67171: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 23:39:16.645925 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.645494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36258)
0810 23:39:16.645647 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.645752 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 23:39:16.646123 11346 consensus_peers.cc:487] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 -> Peer 97a992b3716d4a66b6ca82dd811968fd (127.30.33.62:40579): Couldn't send request to peer 97a992b3716d4a66b6ca82dd811968fd. Status: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:39:16.646150 12437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 23:39:16.649494 12439 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8babcf8bb3647998cf0020b94c67171. Latest consensus state: current_term: 2 leader_uuid: "d8babcf8bb3647998cf0020b94c67171" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:16.649791 12439 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:39:16.651594 12437 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d8babcf8bb3647998cf0020b94c67171" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:16.651940 12437 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:16.652665 12439 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d8babcf8bb3647998cf0020b94c67171" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "97a992b3716d4a66b6ca82dd811968fd" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40579 } } peers { permanent_uuid: "c8029edc52e745609f8f4878a3704ee5" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 41497 } attrs { promote: false } } peers { permanent_uuid: "d8babcf8bb3647998cf0020b94c67171" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 39057 } attrs { promote: false } } }
I20260810 23:39:16.652956 12439 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8029edc52e745609f8f4878a3704ee5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:39:16.680457 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.661690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43999 (local address 127.0.0.1:40514)
0810 23:39:16.672350 (+ 10660us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.672381 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:16.672569 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:16.674094 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:16.674101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:16.674144 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:16.674568 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.674583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.676447 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.676453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.678152 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.678168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.679372 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.679379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.679645 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.679696 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 23:39:16.679810 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:39:16.679962 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10477,"mutex_wait_us":1,"thread_start_us":10241,"threads_started":1}
I20260810 23:39:16.681301 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.662104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.30.33.1:43999)
0810 23:39:16.673589 (+ 11485us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:16.673597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:16.673627 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:39:16.673722 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:16.673728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:16.673821 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:16.673977 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:16.674789 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.676301 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.678304 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.679102 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.679253 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.679371 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:16.680819 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0810 23:39:16.680990 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11321,"thread_start_us":12012,"threads_started":1}
I20260810 23:39:16.682173 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.661084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:44785 (local address 127.0.0.1:50794)
0810 23:39:16.661432 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.661456 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:16.661586 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:16.675032 (+ 13446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:16.675038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:16.675069 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:16.675256 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.675269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.677811 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.677817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.679776 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.679795 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.681377 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.681383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.681609 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.681652 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:39:16.681748 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:39:16.681888 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":21}
I20260810 23:39:16.682507 11608 tablet_service.cc:1467] Processing CreateTablet for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 (DEFAULT_TABLE table=test-table [id=53e1d29215bd447fb5fb7af90de588cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:16.682824 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.661090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.30.33.3:44785)
0810 23:39:16.674659 (+ 13569us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:16.674666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:16.674688 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:39:16.674760 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:16.674765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:16.674857 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:16.674976 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:16.675418 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.677641 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.680035 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.681217 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.681345 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.681442 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:16.682394 (+ 952us) server_negotiation.cc:299] Negotiation successful
0810 23:39:16.682530 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13423,"thread_start_us":13290,"threads_started":1}
I20260810 23:39:16.683033 11608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:16.683811 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.661328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41157 (local address 127.0.0.1:59798)
0810 23:39:16.661882 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.661900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:16.661994 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:16.677461 (+ 15467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:16.677467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:16.677495 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:16.677949 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.677962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.679453 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.679494 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.681654 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.681670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.683162 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.683169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.683306 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.683337 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:39:16.683430 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:39:16.683586 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":32}
I20260810 23:39:16.683951 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.661676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.30.33.2:41157)
0810 23:39:16.677118 (+ 15442us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:16.677124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:16.677144 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:39:16.677218 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:16.677223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:16.677296 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:16.677412 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:16.678157 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.679274 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.681867 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.682933 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.683069 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.683186 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:16.683512 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 23:39:16.683664 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15301,"thread_start_us":15831,"threads_started":1}
I20260810 23:39:16.688717 12477 tablet_bootstrap.cc:492] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: Bootstrap starting.
I20260810 23:39:16.688994 11912 tablet_service.cc:1467] Processing CreateTablet for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 (DEFAULT_TABLE table=test-table [id=53e1d29215bd447fb5fb7af90de588cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:16.689235 12176 tablet_service.cc:1467] Processing CreateTablet for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 (DEFAULT_TABLE table=test-table [id=53e1d29215bd447fb5fb7af90de588cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:16.689522 11912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:16.689677 12176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:16.691428 12477 tablet_bootstrap.cc:654] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:16.693506 12477 tablet_bootstrap.cc:492] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: No bootstrap required, opened a new log
I20260810 23:39:16.693660 12477 ts_tablet_manager.cc:1397] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260810 23:39:16.694352 12477 raft_consensus.cc:348] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.694552 12477 raft_consensus.cc:374] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:16.694643 12477 raft_consensus.cc:729] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3c2f7a295c4fe891e0044311bc9a99, State: Initialized, Role: FOLLOWER
I20260810 23:39:16.694893 12477 consensus_queue.cc:260] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.695766 12477 ts_tablet_manager.cc:1428] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 23:39:16.699211 11749 heartbeater.cc:503] Master 127.30.33.60:39057 was elected leader, sending a full tablet report...
I20260810 23:39:16.703969 12488 tablet_bootstrap.cc:492] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: Bootstrap starting.
I20260810 23:39:16.705394 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.696136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.1:52717)
0810 23:39:16.704810 (+ 8674us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.705001 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8502,"thread_start_us":7521,"threads_started":1}
I20260810 23:39:16.706197 12487 tablet_bootstrap.cc:492] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: Bootstrap starting.
I20260810 23:39:16.707232 12488 tablet_bootstrap.cc:654] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:16.707857 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.707075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.1:54729)
0810 23:39:16.707310 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.707405 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 23:39:16.708302 12487 tablet_bootstrap.cc:654] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:16.709617 12488 tablet_bootstrap.cc:492] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: No bootstrap required, opened a new log
I20260810 23:39:16.709764 12488 ts_tablet_manager.cc:1397] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 23:39:16.710547 12487 tablet_bootstrap.cc:492] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: No bootstrap required, opened a new log
I20260810 23:39:16.710709 12487 ts_tablet_manager.cc:1397] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 23:39:16.710638 12488 raft_consensus.cc:348] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.710902 12488 raft_consensus.cc:374] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:16.711005 12488 raft_consensus.cc:729] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a3eb08c9b5401b99a2dc64b2590c53, State: Initialized, Role: FOLLOWER
I20260810 23:39:16.711278 12488 consensus_queue.cc:260] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.711563 12487 raft_consensus.cc:348] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.711803 12487 raft_consensus.cc:374] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:16.711900 12487 raft_consensus.cc:729] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a41d48a84aa4d34a000ba1a59f213c3, State: Initialized, Role: FOLLOWER
I20260810 23:39:16.712180 12488 ts_tablet_manager.cc:1428] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 23:39:16.712168 12487 consensus_queue.cc:260] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.714179 12487 ts_tablet_manager.cc:1428] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:16.715291 12020 heartbeater.cc:503] Master 127.30.33.60:39057 was elected leader, sending a full tablet report...
I20260810 23:39:16.716095 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.712920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.2:53421)
0810 23:39:16.715618 (+ 2698us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.715792 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2499,"threads_started":1}
I20260810 23:39:16.716620 12311 heartbeater.cc:503] Master 127.30.33.60:39057 was elected leader, sending a full tablet report...
I20260810 23:39:16.717437 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.714864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.3:51349)
0810 23:39:16.716976 (+ 2112us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.717149 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1983,"thread_start_us":1812,"threads_started":1}
I20260810 23:39:16.719146 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.718182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.3:58379)
0810 23:39:16.718913 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.719006 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:39:16.719434 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.718846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.2:34951)
0810 23:39:16.719087 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.719190 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 23:39:16.972162 12482 raft_consensus.cc:482] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:16.972416 12482 raft_consensus.cc:504] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.974047 12482 leader_election.cc:288] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785), c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157)
I20260810 23:39:16.981442 12494 raft_consensus.cc:482] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:16.981683 12494 raft_consensus.cc:504] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.982062 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.974658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:47387 (local address 127.30.33.2:41157)
0810 23:39:16.974883 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:16.974889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:16.975258 (+ 369us) server_negotiation.cc:407] Connection header received
0810 23:39:16.975385 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:16.975389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:16.975449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:16.975552 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:16.976090 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.977069 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.979446 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.980783 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.980947 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.981057 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:16.981649 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 23:39:16.981809 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:39:16.982134 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.974338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41157 (local address 127.30.33.1:47387)
0810 23:39:16.975115 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.975133 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:16.975265 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:16.975597 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:16.975601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:16.975629 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:16.975938 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.975946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.977225 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.977230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.979272 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.979282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.981111 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.981117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.981381 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.981451 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 23:39:16.981548 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:39:16.981706 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":229,"thread_start_us":398,"threads_started":1}
I20260810 23:39:16.982633 12494 leader_election.cc:288] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157), 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:16.982686 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.974312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:44785 (local address 127.30.33.1:36107)
0810 23:39:16.974528 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.974551 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:16.974715 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:16.975183 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:16.975189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:16.975226 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:16.975585 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.975596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.976926 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.976930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.979242 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:16.979266 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.981080 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.981092 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:16.981380 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.981447 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 23:39:16.981550 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:39:16.981888 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":49,"spinlock_wait_cycles":3456}
I20260810 23:39:16.982918 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" is_pre_election: true
I20260810 23:39:16.983211 11947 raft_consensus.cc:2454] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3c2f7a295c4fe891e0044311bc9a99 in term 0.
I20260810 23:39:16.983302 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" is_pre_election: true
I20260810 23:39:16.983574 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.974444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:36107 (local address 127.30.33.3:44785)
0810 23:39:16.974643 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:16.974677 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:16.974701 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:39:16.974953 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:16.974958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:16.975041 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:16.975116 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:16.975765 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.976781 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.979488 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:16.980802 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:16.980947 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:16.981056 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:16.982507 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0810 23:39:16.982663 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:39:16.983575 11654 raft_consensus.cc:2454] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in term 0.
I20260810 23:39:16.983569 11946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" is_pre_election: true
I20260810 23:39:16.983865 11946 raft_consensus.cc:2454] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in term 0.
I20260810 23:39:16.983891 11518 leader_election.cc:302] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3c2f7a295c4fe891e0044311bc9a99, c6a3eb08c9b5401b99a2dc64b2590c53; no voters:
I20260810 23:39:16.984309 12217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" is_pre_election: true
I20260810 23:39:16.984571 12482 raft_consensus.cc:2781] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:16.984567 12078 leader_election.cc:302] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a41d48a84aa4d34a000ba1a59f213c3, 8b3c2f7a295c4fe891e0044311bc9a99, c6a3eb08c9b5401b99a2dc64b2590c53; no voters:
I20260810 23:39:16.984679 12482 raft_consensus.cc:482] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:16.984793 12482 raft_consensus.cc:3037] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:16.984805 12217 raft_consensus.cc:2454] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3c2f7a295c4fe891e0044311bc9a99 in term 0.
I20260810 23:39:16.985049 12494 raft_consensus.cc:2781] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:16.985172 12494 raft_consensus.cc:482] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:16.985276 12494 raft_consensus.cc:3037] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:16.988106 12482 raft_consensus.cc:504] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.988106 12494 raft_consensus.cc:504] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.988865 12482 leader_election.cc:288] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [CANDIDATE]: Term 1 election: Requested vote from peers 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785), c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157)
I20260810 23:39:16.989486 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3c2f7a295c4fe891e0044311bc9a99"
I20260810 23:39:16.989498 11946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53"
I20260810 23:39:16.989634 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53"
I20260810 23:39:16.989715 11946 raft_consensus.cc:3037] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:16.989791 11654 raft_consensus.cc:2379] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a41d48a84aa4d34a000ba1a59f213c3 in current term 1: Already voted for candidate 8b3c2f7a295c4fe891e0044311bc9a99 in this term.
I20260810 23:39:16.990476 12494 leader_election.cc:288] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 election: Requested vote from peers c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157), 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999)
I20260810 23:39:16.990825 12217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fafe7c0d7c6400f8d263b5cee4e4cc0" candidate_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a41d48a84aa4d34a000ba1a59f213c3"
I20260810 23:39:16.991107 12217 raft_consensus.cc:2379] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b3c2f7a295c4fe891e0044311bc9a99 in current term 1: Already voted for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in this term.
I20260810 23:39:16.991806 11519 leader_election.cc:302] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b3c2f7a295c4fe891e0044311bc9a99; no voters: 8a41d48a84aa4d34a000ba1a59f213c3, c6a3eb08c9b5401b99a2dc64b2590c53
I20260810 23:39:16.992260 12482 raft_consensus.cc:2726] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:39:16.992646 11946 raft_consensus.cc:2454] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a41d48a84aa4d34a000ba1a59f213c3 in term 1.
I20260810 23:39:16.993144 12079 leader_election.cc:302] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a41d48a84aa4d34a000ba1a59f213c3, c6a3eb08c9b5401b99a2dc64b2590c53; no voters: 8b3c2f7a295c4fe891e0044311bc9a99
I20260810 23:39:16.993489 12494 raft_consensus.cc:2781] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:16.994022 12494 raft_consensus.cc:686] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 LEADER]: Becoming Leader. State: Replica: 8a41d48a84aa4d34a000ba1a59f213c3, State: Running, Role: LEADER
I20260810 23:39:16.994376 12494 consensus_queue.cc:237] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } }
I20260810 23:39:16.995131 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:16.994668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.3:43693)
0810 23:39:16.994837 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:16.994956 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 23:39:16.997766 11379 catalog_manager.cc:5495] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3). New cstate: current_term: 1 leader_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a41d48a84aa4d34a000ba1a59f213c3" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20260810 23:39:17.055212 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.050224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.30.33.3:44785)
0810 23:39:17.050397 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:17.050402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:17.050505 (+ 103us) server_negotiation.cc:407] Connection header received
0810 23:39:17.050629 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:17.050633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:17.050733 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:17.050822 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:17.051214 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.052041 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.053448 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.053889 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.053992 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:17.054468 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:17.054593 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:17.054710 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:17.054848 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 23:39:17.054936 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:39:17.058946 11654 raft_consensus.cc:1261] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Refusing update from remote peer 8a41d48a84aa4d34a000ba1a59f213c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:17.058969 11946 raft_consensus.cc:1261] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Refusing update from remote peer 8a41d48a84aa4d34a000ba1a59f213c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:17.059775 12501 consensus_queue.cc:1035] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a3eb08c9b5401b99a2dc64b2590c53" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:17.060204 12494 consensus_queue.cc:1035] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3c2f7a295c4fe891e0044311bc9a99" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:17.063107 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.062544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.2:45801)
0810 23:39:17.062703 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.062840 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 23:39:17.063210 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.062577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.1:45227)
0810 23:39:17.062890 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.063020 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260810 23:39:17.069873 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.069333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.2:51315)
0810 23:39:17.069569 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.069680 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 23:39:17.071290 12511 mvcc.cc:205] Tried to move back new op lower bound from 7317115523306610688 to 7317115523052085248. Current Snapshot: MvccSnapshot[applied={T|T < 7317115523306610688}]
I20260810 23:39:17.075352 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.074823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.3:35565)
0810 23:39:17.075039 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.075146 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 23:39:17.076875 11379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:17.077184 11379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:17.077443 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.072115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.30.33.1:53865)
0810 23:39:17.077135 (+ 5020us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.077246 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260810 23:39:17.082790 11379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8babcf8bb3647998cf0020b94c67171: Sending DeleteTablet for 3 replicas of tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0
I20260810 23:39:17.083925 12176 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:17 UTC) from {username='oryx', principal='oryx/127.30.33.60@KRBTEST.COM'} at 127.0.0.1:50794
I20260810 23:39:17.084394 11608 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:17 UTC) from {username='oryx', principal='oryx/127.30.33.60@KRBTEST.COM'} at 127.0.0.1:40514
I20260810 23:39:17.084457 11912 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:17 UTC) from {username='oryx', principal='oryx/127.30.33.60@KRBTEST.COM'} at 127.0.0.1:59798
I20260810 23:39:17.085415 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 11352
I20260810 23:39:17.092526 12522 tablet_replica.cc:330] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: stopping tablet replica
I20260810 23:39:17.092834 12522 raft_consensus.cc:2229] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:39:17.093348 12522 raft_consensus.cc:2258] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:17.094798 12522 ts_tablet_manager.cc:1905] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:17.099402 12525 tablet_replica.cc:330] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: stopping tablet replica
I20260810 23:39:17.099642 12524 tablet_replica.cc:330] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: stopping tablet replica
I20260810 23:39:17.099660 12525 raft_consensus.cc:2229] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:17.099932 12524 raft_consensus.cc:2229] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:17.100036 12525 raft_consensus.cc:2258] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:17.100327 12524 raft_consensus.cc:2258] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:17.101500 12524 ts_tablet_manager.cc:1905] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:17.101661 12525 ts_tablet_manager.cc:1905] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:17.102129 12522 ts_tablet_manager.cc:1918] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:17.102290 12522 log.cc:1196] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/2fafe7c0d7c6400f8d263b5cee4e4cc0
I20260810 23:39:17.102775 12522 ts_tablet_manager.cc:1939] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8a41d48a84aa4d34a000ba1a59f213c3: Deleting consensus metadata
I20260810 23:39:17.104128 11348 catalog_manager.cc:4841] TS 8a41d48a84aa4d34a000ba1a59f213c3 (127.30.33.3:44785): tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 (table test-table [id=53e1d29215bd447fb5fb7af90de588cb]) successfully deleted
I20260810 23:39:17.107957 12524 ts_tablet_manager.cc:1918] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:17.108089 12524 log.cc:1196] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/2fafe7c0d7c6400f8d263b5cee4e4cc0
I20260810 23:39:17.108278 12525 ts_tablet_manager.cc:1918] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:17.108436 12525 log.cc:1196] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/2fafe7c0d7c6400f8d263b5cee4e4cc0
I20260810 23:39:17.108670 12524 ts_tablet_manager.cc:1939] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P 8b3c2f7a295c4fe891e0044311bc9a99: Deleting consensus metadata
I20260810 23:39:17.108918 12525 ts_tablet_manager.cc:1939] T 2fafe7c0d7c6400f8d263b5cee4e4cc0 P c6a3eb08c9b5401b99a2dc64b2590c53: Deleting consensus metadata
I20260810 23:39:17.109923 11346 catalog_manager.cc:4841] TS c6a3eb08c9b5401b99a2dc64b2590c53 (127.30.33.2:41157): tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 (table test-table [id=53e1d29215bd447fb5fb7af90de588cb]) successfully deleted
I20260810 23:39:17.110240 11345 catalog_manager.cc:4841] TS 8b3c2f7a295c4fe891e0044311bc9a99 (127.30.33.1:43999): tablet 2fafe7c0d7c6400f8d263b5cee4e4cc0 (table test-table [id=53e1d29215bd447fb5fb7af90de588cb]) successfully deleted
I20260810 23:39:17.114737 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.086951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.30.33.1:43999)
0810 23:39:17.087173 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:17.087180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:17.097960 (+ 10780us) server_negotiation.cc:407] Connection header received
0810 23:39:17.098567 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:17.098574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:17.098686 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:17.098767 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:17.101580 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.102669 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.104674 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.105401 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.105542 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:17.109035 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:17.109059 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:17.109082 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:17.109124 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:17.112222 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:17.112962 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:17.112968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:17.112971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:17.113134 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:17.113476 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:17.113479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:17.113481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:17.113844 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:17.114084 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:17.114305 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 23:39:17.114488 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:39:17.115437 11736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:17.152415 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.151839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36260)
0810 23:39:17.152047 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.152166 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:39:17.154417 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.153924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36266)
0810 23:39:17.154098 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.154214 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 23:39:17.224947 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 11352
I20260810 23:39:17.250658 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 11772
I20260810 23:39:17.264447 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.251610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.30.33.2:41157)
0810 23:39:17.251779 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:17.251783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:17.251803 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:39:17.251918 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:17.251923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:17.252003 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:17.252094 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:17.253425 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.254205 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.255652 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.256114 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.256241 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:17.258815 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:17.258834 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:17.258852 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:17.258891 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:17.261861 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:17.262618 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:17.262625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:17.262628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:17.262771 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:17.263058 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:17.263061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:17.263063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:17.263413 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:17.263723 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:17.263950 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 23:39:17.264176 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:39:17.265185 12015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:17.361528 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 11772
I20260810 23:39:17.390893 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 12050
I20260810 23:39:17.405511 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.391690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.30.33.3:44785)
0810 23:39:17.391920 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:17.391925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:17.391944 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:17.392042 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:17.392046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:17.392112 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:17.392195 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:17.393776 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.394526 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.396081 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:17.396556 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:17.396663 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:17.399604 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:17.399629 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:17.399647 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:17.399689 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:17.402638 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:17.403388 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:17.403395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:17.403398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:17.403594 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:17.403963 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:17.403966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:17.403969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:17.404383 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:17.404629 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:17.404996 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:39:17.405184 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:39:17.406229 12296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:17.527659 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 12050
I20260810 23:39:17.532565 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.532090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36274)
0810 23:39:17.532318 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.532403 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 23:39:17.533687 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:17.533325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40579 (local address 127.0.0.1:36284)
0810 23:39:17.533496 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:17.533563 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 23:39:17.561337 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 10739
I20260810 23:39:17.562641 11055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:17.680639 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 10739
I20260810 23:39:17.712746 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 10927
I20260810 23:39:17.713923 11440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:17.835794 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 10927
2026-08-10T23:39:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10602 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:39:17 dist-test-slave-pww0 krb5kdc[12544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:39:17 dist-test-slave-pww0 krb5kdc[12544](info): set up 2 sockets
Aug 10 23:39:17 dist-test-slave-pww0 krb5kdc[12544](info): commencing operation
krb5kdc: starting...
W20260810 23:39:19.160158 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:39:19 dist-test-slave-pww0 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab.
Entry for principal oryx/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab.
I20260810 23:39:19.452795 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:39333
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab
--principal=oryx/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:39333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:19.803155 12573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:19.803573 12573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:19.803686 12573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:19.811619 12573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:19.811748 12573 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:39:19.811790 12573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:19.811853 12573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:19.811913 12573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:19.811997 12573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:19.822494 12573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:39333
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.62.keytab
--principal=oryx/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:39333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:19.823923 12573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:19.825282 12573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:19.837971 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:19.838053 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:19.837997 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:19.838792 12573 server_base.cc:1034] running on GCE node
I20260810 23:39:19.839735 12573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:19.841296 12573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:19.842536 12573 hybrid_clock.cc:648] HybridClock initialized: now 1786405159842472 us; error 35 us; skew 500 ppm
I20260810 23:39:19.842908 12573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:19 dist-test-slave-pww0 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405159, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:19.846302 12573 init.cc:377] Logged in from keytab as oryx/127.30.33.62@KRBTEST.COM (short username oryx)
I20260810 23:39:19.848060 12573 webserver.cc:460] Webserver started at http://127.30.33.62:37301/ using document root <none> and password file <none>
I20260810 23:39:19.848794 12573 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:19.848891 12573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:19.849197 12573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:19.852412 12573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "81353e539f0948b0b35dbc74b8307117"
format_stamp: "Formatted at 2026-08-10 23:39:19 on dist-test-slave-pww0"
server_key: "0fb71ab4cc5c62c2d8cf0111081d6ab2"
server_key_iv: "098d65c5db039223cb42d158eb7d11de"
server_key_version: "encryptionkey@0"
I20260810 23:39:19.853255 12573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "81353e539f0948b0b35dbc74b8307117"
format_stamp: "Formatted at 2026-08-10 23:39:19 on dist-test-slave-pww0"
server_key: "0fb71ab4cc5c62c2d8cf0111081d6ab2"
server_key_iv: "098d65c5db039223cb42d158eb7d11de"
server_key_version: "encryptionkey@0"
I20260810 23:39:19.858886 12573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:39:19.862749 12603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:19.864290 12573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:39:19.864480 12573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "81353e539f0948b0b35dbc74b8307117"
format_stamp: "Formatted at 2026-08-10 23:39:19 on dist-test-slave-pww0"
server_key: "0fb71ab4cc5c62c2d8cf0111081d6ab2"
server_key_iv: "098d65c5db039223cb42d158eb7d11de"
server_key_version: "encryptionkey@0"
I20260810 23:39:19.864715 12573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:19.885215 12573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:19.889870 12573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:19.890161 12573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:19.912580 12573 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:39333
I20260810 23:39:19.912608 12715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:39333 every 8 connection(s)
I20260810 23:39:19.914012 12573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:19.918040 12721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:19.922483 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 12573
I20260810 23:39:19.922708 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:19.923578 30852 external_mini_cluster.cc:1428] Setting key 259d309ee67648e8f2e52b3b22374098
I20260810 23:39:19.930763 12721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Bootstrap starting.
I20260810 23:39:19.934756 12721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:19.936298 12721 log.cc:824] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:19.939684 12721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: No bootstrap required, opened a new log
Aug 10 23:39:19 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:19.950555 12721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81353e539f0948b0b35dbc74b8307117" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39333 } }
I20260810 23:39:19.950924 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:19.925896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.30.33.62:39333)
0810 23:39:19.926737 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:19.926749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:19.926796 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:39:19.929017 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:19.929045 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:19.929491 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:19.929856 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:19.931796 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:19.933193 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:19.935061 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:19.936257 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:19.936470 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:19.940876 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:19.940891 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:19.940913 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:19.940963 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:19.945321 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:19.946268 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:19.946280 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:19.946291 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:19.946541 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:19.946973 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:19.946978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:19.946981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:19.947923 (+ 942us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:19.948268 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:19.948787 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 23:39:19.949364 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":389,"threads_started":1}
I20260810 23:39:19.951097 12721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:19.951212 12721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81353e539f0948b0b35dbc74b8307117, State: Initialized, Role: FOLLOWER
I20260810 23:39:19.951928 12721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81353e539f0948b0b35dbc74b8307117" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39333 } }
I20260810 23:39:19.952194 12721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:19.952335 12721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:19.952602 12721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:19.955371 12721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81353e539f0948b0b35dbc74b8307117" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39333 } }
I20260810 23:39:19.955965 12721 leader_election.cc:302] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81353e539f0948b0b35dbc74b8307117; no voters:
I20260810 23:39:19.956620 12721 leader_election.cc:288] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:19.956740 12729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:19.957796 12729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [term 1 LEADER]: Becoming Leader. State: Replica: 81353e539f0948b0b35dbc74b8307117, State: Running, Role: LEADER
I20260810 23:39:19.958328 12729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81353e539f0948b0b35dbc74b8307117" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39333 } }
I20260810 23:39:19.958854 12721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:19.961499 12738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81353e539f0948b0b35dbc74b8307117. Latest consensus state: current_term: 1 leader_uuid: "81353e539f0948b0b35dbc74b8307117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81353e539f0948b0b35dbc74b8307117" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39333 } } }
I20260810 23:39:19.961311 12730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81353e539f0948b0b35dbc74b8307117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81353e539f0948b0b35dbc74b8307117" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39333 } } }
I20260810 23:39:19.961943 12730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:19.963356 12738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:19.964161 12746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:19.968849 12746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:19.978266 12746 catalog_manager.cc:1347] Generated new cluster ID: 59da444d47d54fe6be9b853afa3fd807
I20260810 23:39:19.978408 12746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:19.987793 12746 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:19.989239 12746 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:19.999585 12746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Generated new TSK 0
I20260810 23:39:20.000553 12746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab.
Entry for principal oryx/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab.
I20260810 23:39:20.071732 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39333
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab
--principal=oryx/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:20.430756 12766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:20.431063 12766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:20.431195 12766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:20.438800 12766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:20.438926 12766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:20.439052 12766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:39:20.439145 12766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:20.449506 12766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab
--principal=oryx/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:20.450951 12766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:20.452462 12766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:20.467007 12777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:20.467032 12783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:20.466961 12780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:20.468376 12766 server_base.cc:1034] running on GCE node
I20260810 23:39:20.469189 12766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:20.470468 12766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:20.471729 12766 hybrid_clock.cc:648] HybridClock initialized: now 1786405160471644 us; error 47 us; skew 500 ppm
I20260810 23:39:20.472097 12766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:20 dist-test-slave-pww0 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405160, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:20.475844 12766 init.cc:377] Logged in from keytab as oryx/127.30.33.1@KRBTEST.COM (short username oryx)
I20260810 23:39:20.477589 12766 webserver.cc:460] Webserver started at http://127.30.33.1:46459/ using document root <none> and password file <none>
I20260810 23:39:20.478369 12766 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:20.478489 12766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:20.478845 12766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:20.482391 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "a7390201d0d14fb0a61ed7f270ee038c"
format_stamp: "Formatted at 2026-08-10 23:39:20 on dist-test-slave-pww0"
server_key: "155fd468ff97ccc259e67bc6c47fdfa1"
server_key_iv: "9dd0907cd7be20b4172dac3141086e1c"
server_key_version: "encryptionkey@0"
I20260810 23:39:20.483261 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "a7390201d0d14fb0a61ed7f270ee038c"
format_stamp: "Formatted at 2026-08-10 23:39:20 on dist-test-slave-pww0"
server_key: "155fd468ff97ccc259e67bc6c47fdfa1"
server_key_iv: "9dd0907cd7be20b4172dac3141086e1c"
server_key_version: "encryptionkey@0"
I20260810 23:39:20.488986 12766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:39:20.492908 12794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:20.494475 12766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:39:20.494668 12766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "a7390201d0d14fb0a61ed7f270ee038c"
format_stamp: "Formatted at 2026-08-10 23:39:20 on dist-test-slave-pww0"
server_key: "155fd468ff97ccc259e67bc6c47fdfa1"
server_key_iv: "9dd0907cd7be20b4172dac3141086e1c"
server_key_version: "encryptionkey@0"
I20260810 23:39:20.494904 12766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:20.509346 12766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:20.514110 12766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:20.514407 12766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:20.517383 12766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:20.517496 12766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:20.517661 12766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:20.517808 12766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 10 23:39:20 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405160, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:20.543406 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:20.517969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:56403 (local address 127.30.33.62:39333)
0810 23:39:20.518738 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:20.518743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:20.519891 (+ 1148us) server_negotiation.cc:407] Connection header received
0810 23:39:20.522209 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:20.522215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:20.522300 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:20.522397 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:20.524916 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.525963 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.529252 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.529793 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.529952 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:20.535667 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:20.535673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:20.535676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:20.535716 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:20.539507 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:20.540484 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:20.540489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:20.540491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:20.540665 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:20.541341 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:20.541346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:20.541350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:20.541643 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:20.541777 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:20.542904 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0810 23:39:20.543112 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":579,"threads_started":1}
I20260810 23:39:20.545521 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:20.518698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39333 (local address 127.30.33.1:56403)
0810 23:39:20.519722 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:20.519768 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:20.521906 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:20.522730 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:20.522744 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:20.523612 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:20.524688 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:20.524720 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.526224 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.526231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:20.528979 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:20.529007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.530158 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.530167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:20.530571 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:20.531728 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:20.531770 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:20.535344 (+ 3574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:20.539736 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:20.539748 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:20.539773 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:20.540324 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:20.540818 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:20.540824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:20.540828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:20.541193 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:20.541795 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:20.541806 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:20.542399 (+ 593us) client_negotiation.cc:769] Sending connection context
0810 23:39:20.542763 (+ 364us) client_negotiation.cc:240] Negotiation successful
0810 23:39:20.543516 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":450,"threads_started":1}
I20260810 23:39:20.566809 12766 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:42975
I20260810 23:39:20.566829 13013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:42975 every 8 connection(s)
I20260810 23:39:20.568307 12766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:20.568807 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:20.553071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:39133 (local address 127.30.33.62:39333)
0810 23:39:20.553311 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:20.553318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:20.553848 (+ 530us) server_negotiation.cc:407] Connection header received
0810 23:39:20.554066 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:20.554071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:20.554142 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:20.554220 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:20.555683 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.556485 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.558889 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.559355 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.559493 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:20.562211 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:20.562217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:20.562221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:20.562260 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:20.565607 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:20.566592 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:20.566598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:20.566602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:20.566762 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:20.567191 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:20.567197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:20.567201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:20.567566 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:20.567739 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:20.568163 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 23:39:20.568422 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:39:20.568907 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:20.552974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39333 (local address 127.30.33.1:39133)
0810 23:39:20.553714 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:20.553738 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:20.553933 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:20.554276 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:20.554281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:20.554977 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:20.555435 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:20.555452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.556654 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.556661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:20.558618 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:20.558648 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:20.559593 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:20.559601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:20.559821 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:20.560692 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:20.560715 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:20.561996 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:20.565808 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:20.565814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:20.565818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:20.566457 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:20.566887 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:20.566893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:20.566896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:20.567074 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:20.567687 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:20.567693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:20.567890 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 23:39:20.568062 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 23:39:20.568448 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":385,"threads_started":1}
I20260810 23:39:20.571280 13014 heartbeater.cc:347] Connected to a master server at 127.30.33.62:39333
I20260810 23:39:20.571564 13014 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:20.572365 13014 heartbeater.cc:511] Master 127.30.33.62:39333 requested a full tablet report, sending...
I20260810 23:39:20.572497 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 12766
I20260810 23:39:20.572736 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:39:20.573578 30852 external_mini_cluster.cc:1428] Setting key 3f75fe42d5bde6e873cc51ecee55f58b
I20260810 23:39:20.574728 12637 ts_manager.cc:194] Registered new tserver with Master: a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975)
I20260810 23:39:20.576455 12637 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:56403
WARNING: no policy specified for oryx/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab.
Entry for principal oryx/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab.
I20260810 23:39:20.638761 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39333
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab
--principal=oryx/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:21.000684 13021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:21.000995 13021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:21.001137 13021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:21.008705 13021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:21.008836 13021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:21.008951 13021 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:39:21.009065 13021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:21.019353 13021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab
--principal=oryx/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:21.020818 13021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:21.022287 13021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:21.036432 13028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:21.036691 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:21.036963 13027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:21.037700 13021 server_base.cc:1034] running on GCE node
I20260810 23:39:21.038414 13021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:21.039448 13021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:21.040737 13021 hybrid_clock.cc:648] HybridClock initialized: now 1786405161040645 us; error 56 us; skew 500 ppm
I20260810 23:39:21.041070 13021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405161, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:21.044468 13021 init.cc:377] Logged in from keytab as oryx/127.30.33.2@KRBTEST.COM (short username oryx)
I20260810 23:39:21.046046 13021 webserver.cc:460] Webserver started at http://127.30.33.2:42445/ using document root <none> and password file <none>
I20260810 23:39:21.046823 13021 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:21.046936 13021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:21.047271 13021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:21.050556 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "669cc74c05394844a74f5a45d70212da"
format_stamp: "Formatted at 2026-08-10 23:39:21 on dist-test-slave-pww0"
server_key: "f56273ba5a6068344bdb2ec6e3dddc32"
server_key_iv: "d0a84bc2a1a7f8e6e5884f61107b2540"
server_key_version: "encryptionkey@0"
I20260810 23:39:21.051406 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "669cc74c05394844a74f5a45d70212da"
format_stamp: "Formatted at 2026-08-10 23:39:21 on dist-test-slave-pww0"
server_key: "f56273ba5a6068344bdb2ec6e3dddc32"
server_key_iv: "d0a84bc2a1a7f8e6e5884f61107b2540"
server_key_version: "encryptionkey@0"
I20260810 23:39:21.057016 13021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:39:21.061442 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:21.063238 13021 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 23:39:21.063491 13021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "669cc74c05394844a74f5a45d70212da"
format_stamp: "Formatted at 2026-08-10 23:39:21 on dist-test-slave-pww0"
server_key: "f56273ba5a6068344bdb2ec6e3dddc32"
server_key_iv: "d0a84bc2a1a7f8e6e5884f61107b2540"
server_key_version: "encryptionkey@0"
I20260810 23:39:21.063764 13021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:21.088588 13021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:21.093719 13021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:21.093972 13021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:21.097010 13021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:21.097137 13021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:21.097299 13021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:21.097393 13021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405161, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:21.121775 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.097601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:52255 (local address 127.30.33.62:39333)
0810 23:39:21.098360 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.098365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.099618 (+ 1253us) server_negotiation.cc:407] Connection header received
0810 23:39:21.101896 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.101903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.101996 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.102098 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:21.104426 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.105722 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.108215 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.108762 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.108907 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.113721 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:21.113727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:21.113730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:21.113768 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:21.117425 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.118635 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.118642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.118645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.118825 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.119498 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.119502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.119505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.119923 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:21.120085 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.121103 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.121370 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":551,"threads_started":1}
I20260810 23:39:21.123212 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.098430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39333 (local address 127.30.33.2:52255)
0810 23:39:21.099386 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:21.099430 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:21.101566 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:21.102383 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:21.102392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:21.103174 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:21.104177 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.104207 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.105946 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.105953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.108025 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.108044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.109042 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.109051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.109300 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.110166 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:21.110203 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:21.113430 (+ 3227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:21.117646 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.117663 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.117689 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.118404 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.118949 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.118955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.118958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.119336 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.120074 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:21.120087 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:21.120620 (+ 533us) client_negotiation.cc:769] Sending connection context
0810 23:39:21.120982 (+ 362us) client_negotiation.cc:240] Negotiation successful
0810 23:39:21.121646 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":416,"threads_started":1}
I20260810 23:39:21.147353 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.131412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39333 (local address 127.30.33.2:60149)
0810 23:39:21.132189 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:21.132211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:21.132459 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:21.132831 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:21.132836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:21.133506 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:21.133928 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.133945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.135415 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.135419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.137070 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.137093 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.138348 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.138353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.138467 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.139109 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:21.139126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:21.140234 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:21.143611 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.143617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.143625 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.144304 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.145053 (+ 749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.145058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.145066 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.145263 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.145988 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:21.145993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:21.146262 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 23:39:21.146433 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 23:39:21.146814 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":399,"threads_started":1}
I20260810 23:39:21.148190 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.131387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:60149 (local address 127.30.33.62:39333)
0810 23:39:21.131601 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.131608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.132327 (+ 719us) server_negotiation.cc:407] Connection header received
0810 23:39:21.132609 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.132615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.132687 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.132772 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:21.134121 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.135251 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.137282 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.137924 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.138031 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.140452 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:21.140458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:21.140463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:21.140508 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:21.143439 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.144439 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.144445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.144448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.144638 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.145388 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.145393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.145397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.145876 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:21.146031 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.147742 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.147970 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 23:39:21.155278 13261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:42105 every 8 connection(s)
I20260810 23:39:21.155364 13021 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:42105
I20260810 23:39:21.156735 13021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:21.159139 13262 heartbeater.cc:347] Connected to a master server at 127.30.33.62:39333
I20260810 23:39:21.159384 13262 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:21.160112 13262 heartbeater.cc:511] Master 127.30.33.62:39333 requested a full tablet report, sending...
I20260810 23:39:21.161253 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 13021
I20260810 23:39:21.161448 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:39:21.161716 12637 ts_manager.cc:194] Registered new tserver with Master: 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105)
I20260810 23:39:21.162070 30852 external_mini_cluster.cc:1428] Setting key df485990704a421e61f104ecc9f7f618
I20260810 23:39:21.162879 12637 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:52255
WARNING: no policy specified for oryx/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab.
Entry for principal oryx/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab.
I20260810 23:39:21.224941 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:39333
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab
--principal=oryx/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:21.577958 13267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:21.578275 13267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:21.578418 13267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:39:21.579571 13014 heartbeater.cc:503] Master 127.30.33.62:39333 was elected leader, sending a full tablet report...
W20260810 23:39:21.586176 13267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:21.586319 13267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:21.586459 13267 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:39:21.586583 13267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:21.597155 13267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab
--principal=oryx/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.33.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:21.598462 13267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:21.599928 13267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:21.613365 13280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:21.613370 13285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:21.613401 13287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:21.614040 13267 server_base.cc:1034] running on GCE node
I20260810 23:39:21.614740 13267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:21.615816 13267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:21.617034 13267 hybrid_clock.cc:648] HybridClock initialized: now 1786405161616967 us; error 48 us; skew 500 ppm
I20260810 23:39:21.617326 13267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405161, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:21.620460 13267 init.cc:377] Logged in from keytab as oryx/127.30.33.3@KRBTEST.COM (short username oryx)
I20260810 23:39:21.621913 13267 webserver.cc:460] Webserver started at http://127.30.33.3:41891/ using document root <none> and password file <none>
I20260810 23:39:21.622622 13267 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:21.622733 13267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:21.623049 13267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:21.626073 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "df672b447b394b92a222292262ead3b6"
format_stamp: "Formatted at 2026-08-10 23:39:21 on dist-test-slave-pww0"
server_key: "809ef35a54003f45d700f1dfbd39480e"
server_key_iv: "f3fbe0dcafd0871ece9386dc38f26cec"
server_key_version: "encryptionkey@0"
I20260810 23:39:21.626900 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "df672b447b394b92a222292262ead3b6"
format_stamp: "Formatted at 2026-08-10 23:39:21 on dist-test-slave-pww0"
server_key: "809ef35a54003f45d700f1dfbd39480e"
server_key_iv: "f3fbe0dcafd0871ece9386dc38f26cec"
server_key_version: "encryptionkey@0"
I20260810 23:39:21.632295 13267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:39:21.635914 13309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:21.637370 13267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:39:21.637522 13267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "df672b447b394b92a222292262ead3b6"
format_stamp: "Formatted at 2026-08-10 23:39:21 on dist-test-slave-pww0"
server_key: "809ef35a54003f45d700f1dfbd39480e"
server_key_iv: "f3fbe0dcafd0871ece9386dc38f26cec"
server_key_version: "encryptionkey@0"
I20260810 23:39:21.637728 13267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:21.654282 13267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:21.658751 13267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:21.658996 13267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:21.661908 13267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:21.662010 13267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:21.662158 13267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:21.662221 13267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405161, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:21.687446 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.662669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:57595 (local address 127.30.33.62:39333)
0810 23:39:21.665232 (+ 2563us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.665238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.665264 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:39:21.667347 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.667352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.667433 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.667551 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:21.670565 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.671339 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.674048 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.674656 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.674805 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.679699 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:21.679706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:21.679711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:21.679761 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:21.683269 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.684543 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.684548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.684551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.684726 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.685269 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.685272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.685274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.685563 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:21.685680 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.686771 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.687036 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2435,"thread_start_us":2295,"threads_started":1}
I20260810 23:39:21.689257 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.663363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39333 (local address 127.30.33.3:57595)
0810 23:39:21.664334 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:21.664386 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:21.666741 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:21.668305 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:21.668328 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:21.669246 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:21.670349 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.670380 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.671559 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.671565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.673862 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.673882 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.675142 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.675148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.675400 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.676289 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:21.676331 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:21.679382 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:21.683547 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.683560 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.683586 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.684369 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.684832 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.684837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.684842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.685149 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.685676 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:21.685686 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:21.686233 (+ 547us) client_negotiation.cc:769] Sending connection context
0810 23:39:21.686624 (+ 391us) client_negotiation.cc:240] Negotiation successful
0810 23:39:21.687326 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":394,"threads_started":1}
I20260810 23:39:21.713160 13535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:43231 every 8 connection(s)
I20260810 23:39:21.713204 13267 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:43231
I20260810 23:39:21.714973 13267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:21.717352 13536 heartbeater.cc:347] Connected to a master server at 127.30.33.62:39333
I20260810 23:39:21.717694 13536 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:21.718604 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.702094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:54873 (local address 127.30.33.62:39333)
0810 23:39:21.702310 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.702316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.703087 (+ 771us) server_negotiation.cc:407] Connection header received
0810 23:39:21.703295 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.703301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.703390 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.703497 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:21.704533 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.705645 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.708187 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.708870 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.709023 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.711632 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:21.711638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:21.711641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:21.711678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:21.715163 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.716174 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.716182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.716187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.716390 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.716900 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.716906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.716909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.717367 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:21.717532 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.718145 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.718355 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:39:21.718662 13536 heartbeater.cc:511] Master 127.30.33.62:39333 requested a full tablet report, sending...
I20260810 23:39:21.718844 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.702254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39333 (local address 127.30.33.3:54873)
0810 23:39:21.702949 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:21.702968 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:21.703153 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:21.703545 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:21.703550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:21.704018 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:21.704369 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.704382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.705828 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.705835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.707967 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.707993 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.709127 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.709135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.709350 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.710136 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:21.710168 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:21.711345 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:21.715328 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.715333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.715336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.715977 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.716558 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:21.716563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:21.716567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:21.716744 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:21.717602 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:21.717607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:21.717843 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 23:39:21.718007 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 23:39:21.718391 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":394,"threads_started":1}
I20260810 23:39:21.720145 12637 ts_manager.cc:194] Registered new tserver with Master: df672b447b394b92a222292262ead3b6 (127.30.33.3:43231)
I20260810 23:39:21.720959 12637 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:57595
I20260810 23:39:21.724954 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 13267
I20260810 23:39:21.725122 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:39:21.725540 30852 external_mini_cluster.cc:1428] Setting key aab4d9707e2a156ffd2adbf597136224
I20260810 23:39:21.728669 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:39:21.794672 13327 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:21.783200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.30.33.62:39333)
0810 23:39:21.783411 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.783416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.783741 (+ 325us) server_negotiation.cc:407] Connection header received
0810 23:39:21.783881 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.783885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.783981 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.784084 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:21.785624 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.786407 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.787862 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.788307 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.788416 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.794409 (+ 5993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58182: BlockingRecv error: recv got EOF from 127.0.0.1:58182 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 23:39:21 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:21.812813 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.798758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.30.33.62:39333)
0810 23:39:21.799032 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.799054 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.799267 (+ 213us) server_negotiation.cc:407] Connection header received
0810 23:39:21.799498 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.799504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.799583 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.799689 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:21.801018 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.801773 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.803135 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.803674 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.803786 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.807825 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:21.807832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:21.807836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:21.807886 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:21.810383 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.811073 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.811077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.811080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.811214 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:21.811610 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:21.811614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:21.811616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:21.811897 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:21.812104 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.812391 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.812580 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:39:21.817378 12637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:39:21.824652 12637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:21.860745 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.839005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.30.33.2:42105)
0810 23:39:21.848989 (+ 9984us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.849000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.851909 (+ 2909us) server_negotiation.cc:407] Connection header received
0810 23:39:21.852271 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.852281 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.852643 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.852848 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:21.853823 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.855279 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.857991 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.859076 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.859204 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.859443 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.860203 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.860404 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9747,"thread_start_us":9571,"threads_started":1}
I20260810 23:39:21.860749 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.838343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42105 (local address 127.0.0.1:37632)
0810 23:39:21.851750 (+ 13407us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:21.851769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:21.852041 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:21.852996 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:21.853004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:21.853039 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:21.853624 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.853640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.855448 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.855483 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.857825 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.857839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.859293 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.859300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.859905 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.859977 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 23:39:21.860118 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:39:21.860372 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13279,"thread_start_us":14895,"threads_started":1}
I20260810 23:39:21.862577 13569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.837758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42975 (local address 127.0.0.1:48654)
0810 23:39:21.838572 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:21.838610 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:21.838797 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:21.854502 (+ 15705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:21.854506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:21.854532 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:21.854988 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.855006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.856777 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.856783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.858937 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.858953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.861757 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.861765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.861939 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.861965 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:39:21.862044 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:39:21.862176 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":360,"threads_started":1}
I20260810 23:39:21.862871 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.837980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.30.33.1:42975)
0810 23:39:21.854002 (+ 16022us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.854009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.854034 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:39:21.854113 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.854118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.854300 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.854487 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:21.855252 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.856630 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.859979 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.861474 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.861743 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.861953 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.862106 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.862443 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15872,"thread_start_us":15722,"threads_started":1}
I20260810 23:39:21.863322 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.839377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.30.33.3:43231)
0810 23:39:21.851014 (+ 11637us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:21.851024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:21.853687 (+ 2663us) server_negotiation.cc:407] Connection header received
0810 23:39:21.853989 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:21.853998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:21.854287 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:21.854476 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:21.855533 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.857191 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.859974 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.861439 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.861633 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.861882 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:21.862731 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 23:39:21.862912 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11430,"thread_start_us":11660,"threads_started":1}
I20260810 23:39:21.864091 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:21.838922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43231 (local address 127.0.0.1:32962)
0810 23:39:21.853537 (+ 14615us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:21.853575 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:21.853796 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:21.854825 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:21.854830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:21.854856 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:21.855273 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.855284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.857344 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.857350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.859724 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:21.859738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:21.862302 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:21.862312 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:21.862495 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:21.862527 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:39:21.863612 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0810 23:39:21.863764 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14479,"thread_start_us":12550,"threads_started":1}
I20260810 23:39:21.865921 13141 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbcea3fb7d34990bad2beda26097a4d (DEFAULT_TABLE table=test-table [id=1091c16e9a204518a5c1cdd12cde3ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:21.866762 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbcea3fb7d34990bad2beda26097a4d (DEFAULT_TABLE table=test-table [id=1091c16e9a204518a5c1cdd12cde3ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:21.867624 13141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbcea3fb7d34990bad2beda26097a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:21.867393 13403 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbcea3fb7d34990bad2beda26097a4d (DEFAULT_TABLE table=test-table [id=1091c16e9a204518a5c1cdd12cde3ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:21.868384 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbcea3fb7d34990bad2beda26097a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:21.868788 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbcea3fb7d34990bad2beda26097a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:21.881448 13586 tablet_bootstrap.cc:492] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Bootstrap starting.
I20260810 23:39:21.885236 13586 tablet_bootstrap.cc:654] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:21.886497 13586 log.cc:824] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:21.887769 13592 tablet_bootstrap.cc:492] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Bootstrap starting.
I20260810 23:39:21.888837 13591 tablet_bootstrap.cc:492] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Bootstrap starting.
I20260810 23:39:21.891602 13586 tablet_bootstrap.cc:492] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: No bootstrap required, opened a new log
I20260810 23:39:21.891897 13586 ts_tablet_manager.cc:1397] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260810 23:39:21.892877 13592 tablet_bootstrap.cc:654] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:21.893251 13591 tablet_bootstrap.cc:654] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:21.894328 13592 log.cc:824] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:21.894424 13591 log.cc:824] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:21.897884 13592 tablet_bootstrap.cc:492] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: No bootstrap required, opened a new log
I20260810 23:39:21.897886 13591 tablet_bootstrap.cc:492] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: No bootstrap required, opened a new log
I20260810 23:39:21.898204 13592 ts_tablet_manager.cc:1397] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 23:39:21.898216 13591 ts_tablet_manager.cc:1397] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 23:39:21.900803 13586 raft_consensus.cc:348] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:21.901193 13586 raft_consensus.cc:374] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:21.901296 13586 raft_consensus.cc:729] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df672b447b394b92a222292262ead3b6, State: Initialized, Role: FOLLOWER
I20260810 23:39:21.901911 13586 consensus_queue.cc:260] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:21.903020 13536 heartbeater.cc:503] Master 127.30.33.62:39333 was elected leader, sending a full tablet report...
I20260810 23:39:21.905459 13586 ts_tablet_manager.cc:1428] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260810 23:39:21.908911 13592 raft_consensus.cc:348] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:21.909407 13592 raft_consensus.cc:374] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:21.909512 13592 raft_consensus.cc:729] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669cc74c05394844a74f5a45d70212da, State: Initialized, Role: FOLLOWER
I20260810 23:39:21.910214 13592 consensus_queue.cc:260] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:21.911388 13262 heartbeater.cc:503] Master 127.30.33.62:39333 was elected leader, sending a full tablet report...
I20260810 23:39:21.911741 13591 raft_consensus.cc:348] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:21.912245 13591 raft_consensus.cc:374] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:21.912284 13592 ts_tablet_manager.cc:1428] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260810 23:39:21.912346 13591 raft_consensus.cc:729] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7390201d0d14fb0a61ed7f270ee038c, State: Initialized, Role: FOLLOWER
I20260810 23:39:21.913224 13591 consensus_queue.cc:260] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:21.914948 13591 ts_tablet_manager.cc:1428] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
W20260810 23:39:21.967720 13537 tablet.cc:2369] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:39:22.071869 13015 tablet.cc:2369] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:39:22.114060 13602 raft_consensus.cc:482] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:22.114384 13602 raft_consensus.cc:504] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:22.116392 13602 leader_election.cc:288] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df672b447b394b92a222292262ead3b6 (127.30.33.3:43231), 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105)
I20260810 23:39:22.123518 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.117185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:41597 (local address 127.30.33.2:42105)
0810 23:39:22.117395 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.117401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.117781 (+ 380us) server_negotiation.cc:407] Connection header received
0810 23:39:22.117996 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.118002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.118082 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.118155 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:22.118878 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.119674 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.121741 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.122528 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.122683 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.122794 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.123104 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.123241 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:39:22.123520 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.116801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:56357 (local address 127.30.33.3:43231)
0810 23:39:22.117033 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.117040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.117888 (+ 848us) server_negotiation.cc:407] Connection header received
0810 23:39:22.118052 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.118056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.118120 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.118211 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:22.118922 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.119666 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.121707 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.122492 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.122597 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.122718 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.123089 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.123226 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:39:22.123688 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.116810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43231 (local address 127.30.33.1:56357)
0810 23:39:22.117749 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.117768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.117914 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.118245 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.118249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.118274 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:22.118746 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.118757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.119806 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.119810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.121539 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.121554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.122740 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.122747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.122922 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.122949 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.123029 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.123187 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":133,"thread_start_us":592,"threads_started":1}
I20260810 23:39:22.124118 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.116792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42105 (local address 127.30.33.1:41597)
0810 23:39:22.117628 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.117650 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.117827 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.118210 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.118214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.118240 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:22.118720 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.118734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.119800 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.119804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.121549 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.121562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.122779 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.122784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.122942 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.122966 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.123054 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.123192 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":106,"thread_start_us":418,"threads_started":1}
I20260810 23:39:22.124795 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbcea3fb7d34990bad2beda26097a4d" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6" is_pre_election: true
I20260810 23:39:22.125273 13181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbcea3fb7d34990bad2beda26097a4d" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669cc74c05394844a74f5a45d70212da" is_pre_election: true
I20260810 23:39:22.125461 13446 raft_consensus.cc:2454] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 0.
I20260810 23:39:22.125869 13181 raft_consensus.cc:2454] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 0.
I20260810 23:39:22.126288 12798 leader_election.cc:302] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7390201d0d14fb0a61ed7f270ee038c, df672b447b394b92a222292262ead3b6; no voters:
I20260810 23:39:22.126755 13602 raft_consensus.cc:2781] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:22.126930 13602 raft_consensus.cc:482] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:22.127036 13602 raft_consensus.cc:3037] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.129299 13602 raft_consensus.cc:504] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:22.129937 13602 leader_election.cc:288] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 election: Requested vote from peers df672b447b394b92a222292262ead3b6 (127.30.33.3:43231), 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105)
I20260810 23:39:22.130436 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbcea3fb7d34990bad2beda26097a4d" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6"
I20260810 23:39:22.130514 13181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbcea3fb7d34990bad2beda26097a4d" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669cc74c05394844a74f5a45d70212da"
I20260810 23:39:22.130628 13446 raft_consensus.cc:3037] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.130725 13181 raft_consensus.cc:3037] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.132946 13446 raft_consensus.cc:2454] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 1.
I20260810 23:39:22.133126 13181 raft_consensus.cc:2454] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 1.
I20260810 23:39:22.133430 12798 leader_election.cc:302] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7390201d0d14fb0a61ed7f270ee038c, df672b447b394b92a222292262ead3b6; no voters:
I20260810 23:39:22.133781 13602 raft_consensus.cc:2781] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:22.134227 13602 raft_consensus.cc:686] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 1 LEADER]: Becoming Leader. State: Replica: a7390201d0d14fb0a61ed7f270ee038c, State: Running, Role: LEADER
I20260810 23:39:22.134708 13602 consensus_queue.cc:237] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } }
I20260810 23:39:22.140054 12636 catalog_manager.cc:5495] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c reported cstate change: term changed from 0 to 1, leader changed from <none> to a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1). New cstate: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } health_report { overall_health: UNKNOWN } } }
W20260810 23:39:22.158906 13263 tablet.cc:2369] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:39:22.165005 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.158243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.30.33.1:42975)
0810 23:39:22.158447 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.158454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.158539 (+ 85us) server_negotiation.cc:407] Connection header received
0810 23:39:22.158864 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.158869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.159006 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.159172 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:22.159724 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.160831 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.162295 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.162986 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.163110 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.163607 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.164338 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.164489 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.164622 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.164717 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:39:22.174765 13181 raft_consensus.cc:1261] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Refusing update from remote peer a7390201d0d14fb0a61ed7f270ee038c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:22.174762 13446 raft_consensus.cc:1261] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Refusing update from remote peer a7390201d0d14fb0a61ed7f270ee038c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:22.175839 13606 consensus_queue.cc:1035] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [LEADER]: Connected to new peer: Peer: permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:22.176203 13602 consensus_queue.cc:1035] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [LEADER]: Connected to new peer: Peer: permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:22.187806 13611 mvcc.cc:205] Tried to move back new op lower bound from 7317115544256921600 to 7317115544107843584. Current Snapshot: MvccSnapshot[applied={T|T < 7317115544256921600}]
I20260810 23:39:22.200830 12636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:22.201159 12636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:22.205044 12636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Sending DeleteTablet for 3 replicas of tablet 2fbcea3fb7d34990bad2beda26097a4d
I20260810 23:39:22.206451 13141 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbcea3fb7d34990bad2beda26097a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:22 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:37632
I20260810 23:39:22.206451 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbcea3fb7d34990bad2beda26097a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:22 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:48654
I20260810 23:39:22.206434 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbcea3fb7d34990bad2beda26097a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:22 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260810 23:39:22.208045 12636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:39:22.208798 12636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:22.219815 13634 tablet_replica.cc:330] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: stopping tablet replica
I20260810 23:39:22.220096 13633 tablet_replica.cc:330] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: stopping tablet replica
I20260810 23:39:22.220446 13634 raft_consensus.cc:2229] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:39:22.220724 13134 tablet_service.cc:1467] Processing CreateTablet for tablet f4f457bcc8b7412e9011e029bb53ff9e (DEFAULT_TABLE table=test-table [id=3be6e7a23c2d46979872b113f9fc9d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:22.220885 13402 tablet_service.cc:1467] Processing CreateTablet for tablet f4f457bcc8b7412e9011e029bb53ff9e (DEFAULT_TABLE table=test-table [id=3be6e7a23c2d46979872b113f9fc9d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:22.221288 13634 raft_consensus.cc:2258] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:22.221352 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f457bcc8b7412e9011e029bb53ff9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:22.221375 13134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f457bcc8b7412e9011e029bb53ff9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:22.221205 12888 tablet_service.cc:1467] Processing CreateTablet for tablet f4f457bcc8b7412e9011e029bb53ff9e (DEFAULT_TABLE table=test-table [id=3be6e7a23c2d46979872b113f9fc9d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:22.221740 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f457bcc8b7412e9011e029bb53ff9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:22.228536 13586 tablet_bootstrap.cc:492] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: Bootstrap starting.
I20260810 23:39:22.228914 13632 tablet_replica.cc:330] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: stopping tablet replica
I20260810 23:39:22.229430 13632 raft_consensus.cc:2229] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:22.229568 13633 raft_consensus.cc:2229] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:22.229948 13632 raft_consensus.cc:2258] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:22.230155 13633 raft_consensus.cc:2258] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:22.230392 13586 tablet_bootstrap.cc:654] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:22.231180 13592 tablet_bootstrap.cc:492] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: Bootstrap starting.
I20260810 23:39:22.231822 13634 ts_tablet_manager.cc:1905] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:22.233006 13592 tablet_bootstrap.cc:654] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:22.237766 13633 ts_tablet_manager.cc:1905] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:22.239781 13632 ts_tablet_manager.cc:1905] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:22.244874 13586 tablet_bootstrap.cc:492] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: No bootstrap required, opened a new log
I20260810 23:39:22.245097 13586 ts_tablet_manager.cc:1397] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260810 23:39:22.245167 13633 ts_tablet_manager.cc:1918] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:22.245350 13633 log.cc:1196] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/2fbcea3fb7d34990bad2beda26097a4d
I20260810 23:39:22.245903 13633 ts_tablet_manager.cc:1939] T 2fbcea3fb7d34990bad2beda26097a4d P 669cc74c05394844a74f5a45d70212da: Deleting consensus metadata
I20260810 23:39:22.245956 13586 raft_consensus.cc:348] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.246233 13586 raft_consensus.cc:374] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:22.246336 13586 raft_consensus.cc:729] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df672b447b394b92a222292262ead3b6, State: Initialized, Role: FOLLOWER
I20260810 23:39:22.246639 13586 consensus_queue.cc:260] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.248052 12607 catalog_manager.cc:4841] TS 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105): tablet 2fbcea3fb7d34990bad2beda26097a4d (table test-table [id=1091c16e9a204518a5c1cdd12cde3ed0]) successfully deleted
I20260810 23:39:22.249308 13592 tablet_bootstrap.cc:492] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: No bootstrap required, opened a new log
I20260810 23:39:22.249472 13592 ts_tablet_manager.cc:1397] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260810 23:39:22.249752 13634 ts_tablet_manager.cc:1918] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:22.249842 13586 ts_tablet_manager.cc:1428] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260810 23:39:22.249948 13634 log.cc:1196] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/2fbcea3fb7d34990bad2beda26097a4d
I20260810 23:39:22.250337 13592 raft_consensus.cc:348] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.250569 13634 ts_tablet_manager.cc:1939] T 2fbcea3fb7d34990bad2beda26097a4d P a7390201d0d14fb0a61ed7f270ee038c: Deleting consensus metadata
I20260810 23:39:22.250597 13592 raft_consensus.cc:374] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:22.250710 13592 raft_consensus.cc:729] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669cc74c05394844a74f5a45d70212da, State: Initialized, Role: FOLLOWER
I20260810 23:39:22.250989 13592 consensus_queue.cc:260] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.251682 13592 ts_tablet_manager.cc:1428] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:39:22.252452 12610 catalog_manager.cc:4841] TS a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975): tablet 2fbcea3fb7d34990bad2beda26097a4d (table test-table [id=1091c16e9a204518a5c1cdd12cde3ed0]) successfully deleted
I20260810 23:39:22.252694 13591 tablet_bootstrap.cc:492] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: Bootstrap starting.
I20260810 23:39:22.254293 13591 tablet_bootstrap.cc:654] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:22.254824 13632 ts_tablet_manager.cc:1918] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:22.255013 13632 log.cc:1196] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/2fbcea3fb7d34990bad2beda26097a4d
I20260810 23:39:22.255576 13632 ts_tablet_manager.cc:1939] T 2fbcea3fb7d34990bad2beda26097a4d P df672b447b394b92a222292262ead3b6: Deleting consensus metadata
I20260810 23:39:22.257334 12610 catalog_manager.cc:4841] TS df672b447b394b92a222292262ead3b6 (127.30.33.3:43231): tablet 2fbcea3fb7d34990bad2beda26097a4d (table test-table [id=1091c16e9a204518a5c1cdd12cde3ed0]) successfully deleted
I20260810 23:39:22.257427 13591 tablet_bootstrap.cc:492] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: No bootstrap required, opened a new log
I20260810 23:39:22.257591 13591 ts_tablet_manager.cc:1397] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 23:39:22.258474 13591 raft_consensus.cc:348] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.258723 13591 raft_consensus.cc:374] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:22.258828 13591 raft_consensus.cc:729] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7390201d0d14fb0a61ed7f270ee038c, State: Initialized, Role: FOLLOWER
I20260810 23:39:22.259115 13591 consensus_queue.cc:260] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.259807 13591 ts_tablet_manager.cc:1428] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:39:22.604691 13599 raft_consensus.cc:482] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:22.604943 13599 raft_consensus.cc:504] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.606560 13599 leader_election.cc:288] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975), 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105)
I20260810 23:39:22.614516 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.607158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:36155 (local address 127.30.33.1:42975)
0810 23:39:22.607385 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.607390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.608063 (+ 673us) server_negotiation.cc:407] Connection header received
0810 23:39:22.608300 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.608305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.608388 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.608476 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:22.609209 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.610300 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.612090 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.613340 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.613506 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.613616 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.613989 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.614161 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:39:22.614599 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.607226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:52099 (local address 127.30.33.2:42105)
0810 23:39:22.607579 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.607585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.607872 (+ 287us) server_negotiation.cc:407] Connection header received
0810 23:39:22.608112 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.608118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.608222 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.608333 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:22.608902 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.609937 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.612322 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.613497 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.613607 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.613711 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.614178 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.614316 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260810 23:39:22.615442 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.606901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42975 (local address 127.30.33.3:36155)
0810 23:39:22.607973 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.607989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.608147 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.608538 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.608543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.608572 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:22.608992 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.609004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.610427 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.610431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.611929 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.611944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.613557 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.613564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.613761 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.613802 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.614818 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.614971 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":176,"thread_start_us":609,"threads_started":1}
I20260810 23:39:22.615497 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.606888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42105 (local address 127.30.33.3:52099)
0810 23:39:22.607730 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.607751 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.607931 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.608354 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.608360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.608383 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:22.608732 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.608747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.610079 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.610084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.612151 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.612168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.613796 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.613802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.613969 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.614000 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.614920 (+ 920us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.615061 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":1,"thread_start_us":471,"threads_started":1}
I20260810 23:39:22.616281 13181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f457bcc8b7412e9011e029bb53ff9e" candidate_uuid: "df672b447b394b92a222292262ead3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669cc74c05394844a74f5a45d70212da" is_pre_election: true
I20260810 23:39:22.616580 13181 raft_consensus.cc:2454] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df672b447b394b92a222292262ead3b6 in term 0.
I20260810 23:39:22.616473 12923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f457bcc8b7412e9011e029bb53ff9e" candidate_uuid: "df672b447b394b92a222292262ead3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7390201d0d14fb0a61ed7f270ee038c" is_pre_election: true
I20260810 23:39:22.616989 12923 raft_consensus.cc:2454] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df672b447b394b92a222292262ead3b6 in term 0.
I20260810 23:39:22.617188 13312 leader_election.cc:302] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, df672b447b394b92a222292262ead3b6; no voters:
I20260810 23:39:22.617676 13599 raft_consensus.cc:2781] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:22.617811 13599 raft_consensus.cc:482] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:22.617906 13599 raft_consensus.cc:3037] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.620178 13599 raft_consensus.cc:504] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.620829 13599 leader_election.cc:288] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [CANDIDATE]: Term 1 election: Requested vote from peers a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975), 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105)
I20260810 23:39:22.621315 12923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f457bcc8b7412e9011e029bb53ff9e" candidate_uuid: "df672b447b394b92a222292262ead3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7390201d0d14fb0a61ed7f270ee038c"
I20260810 23:39:22.621436 13181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f457bcc8b7412e9011e029bb53ff9e" candidate_uuid: "df672b447b394b92a222292262ead3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669cc74c05394844a74f5a45d70212da"
I20260810 23:39:22.621534 12923 raft_consensus.cc:3037] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.621663 13181 raft_consensus.cc:3037] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.623749 13181 raft_consensus.cc:2454] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df672b447b394b92a222292262ead3b6 in term 1.
I20260810 23:39:22.624411 13312 leader_election.cc:302] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, df672b447b394b92a222292262ead3b6; no voters:
I20260810 23:39:22.624671 13599 raft_consensus.cc:2781] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:22.625078 13599 raft_consensus.cc:686] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 1 LEADER]: Becoming Leader. State: Replica: df672b447b394b92a222292262ead3b6, State: Running, Role: LEADER
I20260810 23:39:22.625149 12923 raft_consensus.cc:2454] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df672b447b394b92a222292262ead3b6 in term 1.
I20260810 23:39:22.625552 13599 consensus_queue.cc:237] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.629213 12637 catalog_manager.cc:5495] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to df672b447b394b92a222292262ead3b6 (127.30.33.3). New cstate: current_term: 1 leader_uuid: "df672b447b394b92a222292262ead3b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } health_report { overall_health: HEALTHY } } }
Aug 10 23:39:22 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405159, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.62@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:22.741642 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.726435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:39333 (local address 127.0.0.1:58198)
0810 23:39:22.727409 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.727429 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.727609 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.727994 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.727999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.728896 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:22.729136 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.729148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.730321 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.730327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.731883 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.731900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.732670 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.732676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.732868 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.733550 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:22.733568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:22.735962 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:22.738902 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:22.738917 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:22.738923 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:22.739412 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:22.739893 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:22.739897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:22.739901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:22.740092 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:22.740636 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:22.740643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:22.740839 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.741025 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.741306 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":641,"threads_started":1}
I20260810 23:39:22.741746 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.726483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.30.33.62:39333)
0810 23:39:22.727234 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.727241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.727536 (+ 295us) server_negotiation.cc:407] Connection header received
0810 23:39:22.727725 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.727729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.727805 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.727903 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:22.729347 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.730147 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.732056 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.732490 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.732599 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.736167 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:22.736174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:22.736177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:22.736211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:22.738743 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:22.739587 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:22.739602 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:22.739606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:22.739769 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:22.740228 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:22.740232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:22.740234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:22.740529 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:22.740647 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.741146 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.741431 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":582,"thread_start_us":355,"threads_started":1}
I20260810 23:39:22.747057 12637 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:58198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:39:22.748252 12637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:22.755502 12888 tablet_service.cc:1467] Processing CreateTablet for tablet f65231688bf343939fa224d86f2da3b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2ff967cf56b4cd8bf97c42f09407f08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:39:22.755867 13403 tablet_service.cc:1467] Processing CreateTablet for tablet f65231688bf343939fa224d86f2da3b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2ff967cf56b4cd8bf97c42f09407f08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:39:22.755867 13141 tablet_service.cc:1467] Processing CreateTablet for tablet f65231688bf343939fa224d86f2da3b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2ff967cf56b4cd8bf97c42f09407f08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:39:22.756129 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65231688bf343939fa224d86f2da3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:22.756418 13141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65231688bf343939fa224d86f2da3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:22.756439 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65231688bf343939fa224d86f2da3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:22.762269 13662 tablet_bootstrap.cc:492] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: Bootstrap starting.
I20260810 23:39:22.763839 13662 tablet_bootstrap.cc:654] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:22.766047 13662 tablet_bootstrap.cc:492] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: No bootstrap required, opened a new log
I20260810 23:39:22.766609 13662 ts_tablet_manager.cc:1397] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:39:22.767374 13662 raft_consensus.cc:348] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.767688 13662 raft_consensus.cc:374] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:22.767800 13662 raft_consensus.cc:729] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7390201d0d14fb0a61ed7f270ee038c, State: Initialized, Role: FOLLOWER
I20260810 23:39:22.768055 13662 consensus_queue.cc:260] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.767935 13665 tablet_bootstrap.cc:492] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: Bootstrap starting.
I20260810 23:39:22.768728 13664 tablet_bootstrap.cc:492] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: Bootstrap starting.
I20260810 23:39:22.770265 13665 tablet_bootstrap.cc:654] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:22.770532 13664 tablet_bootstrap.cc:654] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:22.772068 13662 ts_tablet_manager.cc:1428] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:39:22.772590 13665 tablet_bootstrap.cc:492] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: No bootstrap required, opened a new log
I20260810 23:39:22.772796 13665 ts_tablet_manager.cc:1397] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 23:39:22.772684 13667 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.773037 13667 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:39:22.773617 13665 raft_consensus.cc:348] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.773903 13665 raft_consensus.cc:374] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:22.773998 13665 raft_consensus.cc:729] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669cc74c05394844a74f5a45d70212da, State: Initialized, Role: FOLLOWER
I20260810 23:39:22.774030 13664 tablet_bootstrap.cc:492] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: No bootstrap required, opened a new log
I20260810 23:39:22.774190 13664 ts_tablet_manager.cc:1397] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260810 23:39:22.774287 13665 consensus_queue.cc:260] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.774947 13664 raft_consensus.cc:348] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.775207 13664 raft_consensus.cc:374] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:22.775226 13665 ts_tablet_manager.cc:1428] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 23:39:22.775300 13664 raft_consensus.cc:729] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df672b447b394b92a222292262ead3b6, State: Initialized, Role: FOLLOWER
I20260810 23:39:22.775589 13664 consensus_queue.cc:260] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.776011 13599 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.776347 13599 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:39:22.776007 13670 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.776602 13670 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:39:22.777295 13664 ts_tablet_manager.cc:1428] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 23:39:22.786036 13667 raft_consensus.cc:482] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:22.786259 13667 raft_consensus.cc:504] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.786913 13667 leader_election.cc:288] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105), df672b447b394b92a222292262ead3b6 (127.30.33.3:43231)
I20260810 23:39:22.787492 13181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65231688bf343939fa224d86f2da3b2" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669cc74c05394844a74f5a45d70212da" is_pre_election: true
I20260810 23:39:22.787510 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65231688bf343939fa224d86f2da3b2" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6" is_pre_election: true
I20260810 23:39:22.787755 13446 raft_consensus.cc:2454] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 0.
I20260810 23:39:22.787778 13181 raft_consensus.cc:2454] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 0.
I20260810 23:39:22.788204 12798 leader_election.cc:302] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, a7390201d0d14fb0a61ed7f270ee038c, df672b447b394b92a222292262ead3b6; no voters:
I20260810 23:39:22.788507 13667 raft_consensus.cc:2781] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:22.788636 13667 raft_consensus.cc:482] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:22.788735 13667 raft_consensus.cc:3037] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.790974 13667 raft_consensus.cc:504] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.791621 13667 leader_election.cc:288] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 election: Requested vote from peers 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105), df672b447b394b92a222292262ead3b6 (127.30.33.3:43231)
I20260810 23:39:22.792168 13181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65231688bf343939fa224d86f2da3b2" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669cc74c05394844a74f5a45d70212da"
I20260810 23:39:22.792173 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65231688bf343939fa224d86f2da3b2" candidate_uuid: "a7390201d0d14fb0a61ed7f270ee038c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6"
I20260810 23:39:22.792411 13181 raft_consensus.cc:3037] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.792412 13446 raft_consensus.cc:3037] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:22.794474 13181 raft_consensus.cc:2454] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 1.
I20260810 23:39:22.794493 13446 raft_consensus.cc:2454] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7390201d0d14fb0a61ed7f270ee038c in term 1.
I20260810 23:39:22.794961 12798 leader_election.cc:302] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, a7390201d0d14fb0a61ed7f270ee038c, df672b447b394b92a222292262ead3b6; no voters:
I20260810 23:39:22.795275 13667 raft_consensus.cc:2781] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:22.795791 13667 raft_consensus.cc:686] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [term 1 LEADER]: Becoming Leader. State: Replica: a7390201d0d14fb0a61ed7f270ee038c, State: Running, Role: LEADER
I20260810 23:39:22.796098 13667 consensus_queue.cc:237] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:22.797961 13675 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: TxnStatusTablet state changed. Reason: New leader a7390201d0d14fb0a61ed7f270ee038c. Latest consensus state: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.798309 13675 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:39:22.798982 13678 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:39:22.799216 13678 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:39:22.799299 12637 catalog_manager.cc:5495] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c reported cstate change: term changed from 0 to 1, leader changed from <none> to a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1). New cstate: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20260810 23:39:22.819126 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.813190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42975 (local address 127.0.0.1:48660)
0810 23:39:22.813432 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.813453 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.813585 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.814013 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.814016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.814025 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:39:22.814187 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.814206 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.815788 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.815797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.816960 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.816977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.817991 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.817996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.818150 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.818228 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.818674 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.818690 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.818751 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.818880 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:39:22.819200 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.813227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.30.33.1:42975)
0810 23:39:22.813437 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.813443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.813523 (+ 80us) server_negotiation.cc:407] Connection header received
0810 23:39:22.813782 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.813790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.813888 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.814001 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:22.814330 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.815599 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.817147 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.817812 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.817954 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.818419 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.818571 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.818661 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.818795 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.818889 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:39:22.823690 13446 raft_consensus.cc:1261] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Refusing update from remote peer a7390201d0d14fb0a61ed7f270ee038c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:22.823716 13181 raft_consensus.cc:1261] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Refusing update from remote peer a7390201d0d14fb0a61ed7f270ee038c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:22.824400 13675 consensus_queue.cc:1035] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [LEADER]: Connected to new peer: Peer: permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:22.824656 13667 consensus_queue.cc:1035] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c [LEADER]: Connected to new peer: Peer: permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:22.832542 13670 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: TxnStatusTablet state changed. Reason: New leader a7390201d0d14fb0a61ed7f270ee038c. Latest consensus state: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.832887 13670 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:39:22.835521 13599 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: TxnStatusTablet state changed. Reason: New leader a7390201d0d14fb0a61ed7f270ee038c. Latest consensus state: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.835857 13599 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:39:22.837133 13667 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.837499 13667 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:39:22.840261 13693 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.840688 13693 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P a7390201d0d14fb0a61ed7f270ee038c: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:39:22.840888 13599 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.841188 13599 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P df672b447b394b92a222292262ead3b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:39:22.841871 13685 mvcc.cc:205] Tried to move back new op lower bound from 7317115546918912000 to 7317115546815942656. Current Snapshot: MvccSnapshot[applied={T|T < 7317115546918912000}]
I20260810 23:39:22.848726 13670 tablet_replica.cc:439] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7390201d0d14fb0a61ed7f270ee038c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } } }
I20260810 23:39:22.849047 13670 tablet_replica.cc:442] T f65231688bf343939fa224d86f2da3b2 P 669cc74c05394844a74f5a45d70212da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:39:22.860018 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.852707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.30.33.3:43231)
0810 23:39:22.853452 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.853457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.853480 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:39:22.853574 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.853578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.853657 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.853804 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:22.854420 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.855233 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.857287 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.857765 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.857874 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.858410 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.859116 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.859238 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.859383 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.859495 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":484,"threads_started":1}
I20260810 23:39:22.878890 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.872509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42975 (local address 127.30.33.3:37555)
0810 23:39:22.873203 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.873221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.873370 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.873722 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.873725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.873734 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:39:22.874017 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.874025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.875556 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.875560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.876653 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.876667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.877735 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.877740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.877874 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.877942 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.878387 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.878398 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.878461 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.878578 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":361,"threads_started":1}
I20260810 23:39:22.878926 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.872522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:37555 (local address 127.30.33.1:42975)
0810 23:39:22.872748 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.872756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.873305 (+ 549us) server_negotiation.cc:407] Connection header received
0810 23:39:22.873501 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.873507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.873602 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.873716 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:22.874158 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.875383 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.876808 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.877541 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.877675 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.878118 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.878289 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.878381 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.878501 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.878591 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:39:22.889424 13181 raft_consensus.cc:1261] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Refusing update from remote peer df672b447b394b92a222292262ead3b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:22.889776 12923 raft_consensus.cc:1261] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Refusing update from remote peer df672b447b394b92a222292262ead3b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:22.890266 13644 consensus_queue.cc:1035] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:22.890755 13599 consensus_queue.cc:1035] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:22.897437 13690 mvcc.cc:205] Tried to move back new op lower bound from 7317115547188011008 to 7317115546118356992. Current Snapshot: MvccSnapshot[applied={T|T < 7317115547188011008}]
I20260810 23:39:22.945969 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.930037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:34153 (local address 127.30.33.3:43231)
0810 23:39:22.930239 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:22.930257 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:22.939061 (+ 8804us) server_negotiation.cc:407] Connection header received
0810 23:39:22.939285 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:22.939292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:22.939389 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:22.939521 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:22.940133 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.941297 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.943671 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.944347 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.944444 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.945171 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.945322 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.945421 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:22.945579 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:39:22.945674 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:39:22.946527 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:22.929842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43231 (local address 127.30.33.1:34153)
0810 23:39:22.938906 (+ 9064us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:22.938935 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:22.939114 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:22.939549 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:22.939554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:22.939573 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:39:22.939966 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.939978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.941475 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.941481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.943485 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:22.943504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:22.944645 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:22.944651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:22.944867 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:22.944954 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:39:22.945448 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:39:22.945462 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:39:22.945546 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:39:22.945734 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8779,"thread_start_us":8607,"threads_started":1}
I20260810 23:39:22.999832 12632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:23.000130 12632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:23.004570 12632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Sending DeleteTablet for 3 replicas of tablet f4f457bcc8b7412e9011e029bb53ff9e
I20260810 23:39:23.005916 13141 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f457bcc8b7412e9011e029bb53ff9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:22 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:37632
I20260810 23:39:23.006493 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f457bcc8b7412e9011e029bb53ff9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:22 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:48654
I20260810 23:39:23.007282 13730 tablet_replica.cc:330] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: stopping tablet replica
I20260810 23:39:23.007495 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f457bcc8b7412e9011e029bb53ff9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:22 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260810 23:39:23.007591 13730 raft_consensus.cc:2229] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:23.008057 13730 raft_consensus.cc:2258] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.009512 13730 ts_tablet_manager.cc:1905] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.011490 13731 tablet_replica.cc:330] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: stopping tablet replica
I20260810 23:39:23.011723 13731 raft_consensus.cc:2229] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:39:23.012430 13731 raft_consensus.cc:2258] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.013047 13728 tablet_replica.cc:330] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: stopping tablet replica
I20260810 23:39:23.013255 13728 raft_consensus.cc:2229] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:23.013517 13728 raft_consensus.cc:2258] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.014638 13728 ts_tablet_manager.cc:1905] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.021588 13728 ts_tablet_manager.cc:1918] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:39:23.021751 13728 log.cc:1196] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/f4f457bcc8b7412e9011e029bb53ff9e
I20260810 23:39:23.022213 13728 ts_tablet_manager.cc:1939] T f4f457bcc8b7412e9011e029bb53ff9e P 669cc74c05394844a74f5a45d70212da: Deleting consensus metadata
I20260810 23:39:23.023530 12607 catalog_manager.cc:4841] TS 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105): tablet f4f457bcc8b7412e9011e029bb53ff9e (table test-table [id=3be6e7a23c2d46979872b113f9fc9d36]) successfully deleted
I20260810 23:39:23.024014 13731 ts_tablet_manager.cc:1905] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.031340 13730 ts_tablet_manager.cc:1918] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:39:23.031570 13731 ts_tablet_manager.cc:1918] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:39:23.031741 13731 log.cc:1196] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/f4f457bcc8b7412e9011e029bb53ff9e
I20260810 23:39:23.031780 13730 log.cc:1196] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/f4f457bcc8b7412e9011e029bb53ff9e
I20260810 23:39:23.032231 13731 ts_tablet_manager.cc:1939] T f4f457bcc8b7412e9011e029bb53ff9e P df672b447b394b92a222292262ead3b6: Deleting consensus metadata
I20260810 23:39:23.032395 13730 ts_tablet_manager.cc:1939] T f4f457bcc8b7412e9011e029bb53ff9e P a7390201d0d14fb0a61ed7f270ee038c: Deleting consensus metadata
I20260810 23:39:23.033438 12610 catalog_manager.cc:4841] TS df672b447b394b92a222292262ead3b6 (127.30.33.3:43231): tablet f4f457bcc8b7412e9011e029bb53ff9e (table test-table [id=3be6e7a23c2d46979872b113f9fc9d36]) successfully deleted
I20260810 23:39:23.033735 12610 catalog_manager.cc:4841] TS a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975): tablet f4f457bcc8b7412e9011e029bb53ff9e (table test-table [id=3be6e7a23c2d46979872b113f9fc9d36]) successfully deleted
Aug 10 23:39:23 dist-test-slave-pww0 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:39:23 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:23 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:39:23.089203 13660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:23.077047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.30.33.62:39333)
0810 23:39:23.077278 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.077285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.077795 (+ 510us) server_negotiation.cc:407] Connection header received
0810 23:39:23.078045 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.078050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.078129 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.078230 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:23.079846 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.080642 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.082138 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.082844 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.083023 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.088963 (+ 5940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58214: BlockingRecv error: recv got EOF from 127.0.0.1:58214 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 23:39:23 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM
I20260810 23:39:23.109009 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.093361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.30.33.62:39333)
0810 23:39:23.093583 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.093591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.094058 (+ 467us) server_negotiation.cc:407] Connection header received
0810 23:39:23.094260 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.094266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.094341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.094429 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:23.095709 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.096482 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.097790 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.098249 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.098371 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.103227 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:23.103233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:23.103236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:23.103279 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:23.106032 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:23.106825 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:23.106831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:23.106833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:23.106991 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:23.107413 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:23.107417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:23.107419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:23.107817 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:23.108093 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.108393 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.108641 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:39:23.113023 12632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:39:23.113705 12632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:23.121371 13141 tablet_service.cc:1467] Processing CreateTablet for tablet ad4ded065e71426ca88b56ea59023aab (DEFAULT_TABLE table=test-table [id=68c6eb94a8f34b089d9bbd201eb13438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:23.121837 13141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4ded065e71426ca88b56ea59023aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:23.125672 12888 tablet_service.cc:1467] Processing CreateTablet for tablet ad4ded065e71426ca88b56ea59023aab (DEFAULT_TABLE table=test-table [id=68c6eb94a8f34b089d9bbd201eb13438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:23.126297 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4ded065e71426ca88b56ea59023aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:23.128595 13403 tablet_service.cc:1467] Processing CreateTablet for tablet ad4ded065e71426ca88b56ea59023aab (DEFAULT_TABLE table=test-table [id=68c6eb94a8f34b089d9bbd201eb13438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:23.129091 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4ded065e71426ca88b56ea59023aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:23.142902 13662 tablet_bootstrap.cc:492] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: Bootstrap starting.
I20260810 23:39:23.144487 13664 tablet_bootstrap.cc:492] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: Bootstrap starting.
I20260810 23:39:23.145134 13662 tablet_bootstrap.cc:654] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:23.146328 13664 tablet_bootstrap.cc:654] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:23.148753 13665 tablet_bootstrap.cc:492] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: Bootstrap starting.
I20260810 23:39:23.150708 13665 tablet_bootstrap.cc:654] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:23.151036 13662 tablet_bootstrap.cc:492] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: No bootstrap required, opened a new log
I20260810 23:39:23.151238 13662 ts_tablet_manager.cc:1397] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 23:39:23.153947 13662 raft_consensus.cc:348] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.154297 13662 raft_consensus.cc:374] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:23.154381 13662 raft_consensus.cc:729] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7390201d0d14fb0a61ed7f270ee038c, State: Initialized, Role: FOLLOWER
I20260810 23:39:23.167526 13662 consensus_queue.cc:260] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.168882 13665 tablet_bootstrap.cc:492] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: No bootstrap required, opened a new log
I20260810 23:39:23.169095 13665 ts_tablet_manager.cc:1397] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20260810 23:39:23.169931 13665 raft_consensus.cc:348] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.171085 13665 raft_consensus.cc:374] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:23.171216 13665 raft_consensus.cc:729] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669cc74c05394844a74f5a45d70212da, State: Initialized, Role: FOLLOWER
I20260810 23:39:23.171541 13665 consensus_queue.cc:260] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.172266 13665 ts_tablet_manager.cc:1428] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:23.174136 13664 tablet_bootstrap.cc:492] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: No bootstrap required, opened a new log
I20260810 23:39:23.174348 13664 ts_tablet_manager.cc:1397] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.012s
I20260810 23:39:23.175179 13664 raft_consensus.cc:348] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.177295 13664 raft_consensus.cc:374] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:23.177423 13664 raft_consensus.cc:729] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df672b447b394b92a222292262ead3b6, State: Initialized, Role: FOLLOWER
I20260810 23:39:23.177747 13664 consensus_queue.cc:260] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.179014 13664 ts_tablet_manager.cc:1428] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:39:23.180487 13662 ts_tablet_manager.cc:1428] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
I20260810 23:39:23.238924 13670 raft_consensus.cc:482] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:23.239245 13670 raft_consensus.cc:504] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.241596 13670 leader_election.cc:288] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975), df672b447b394b92a222292262ead3b6 (127.30.33.3:43231)
I20260810 23:39:23.251060 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.242385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:52751 (local address 127.30.33.1:42975)
0810 23:39:23.242610 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.242617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.243218 (+ 601us) server_negotiation.cc:407] Connection header received
0810 23:39:23.243589 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.243595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.243687 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.243759 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:23.244616 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.245809 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.248455 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.249761 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.249868 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.249973 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.250585 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.250752 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:39:23.252113 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.241921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42975 (local address 127.30.33.2:52751)
0810 23:39:23.243083 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:23.243107 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:23.243362 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:23.244065 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:23.244070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:23.244110 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:23.244404 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.244419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.246035 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.246041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.248247 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.248266 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.250161 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.250168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.250372 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.250409 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:39:23.251606 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0810 23:39:23.251775 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":817,"threads_started":1}
I20260810 23:39:23.255079 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ded065e71426ca88b56ea59023aab" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7390201d0d14fb0a61ed7f270ee038c" is_pre_election: true
I20260810 23:39:23.255424 12921 raft_consensus.cc:2454] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 0.
I20260810 23:39:23.256739 13057 leader_election.cc:302] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, a7390201d0d14fb0a61ed7f270ee038c; no voters:
I20260810 23:39:23.257335 13670 raft_consensus.cc:2781] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:23.257472 13670 raft_consensus.cc:482] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:23.257577 13670 raft_consensus.cc:3037] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:23.261183 13670 raft_consensus.cc:504] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.262020 13670 leader_election.cc:288] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 election: Requested vote from peers a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975), df672b447b394b92a222292262ead3b6 (127.30.33.3:43231)
I20260810 23:39:23.262940 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ded065e71426ca88b56ea59023aab" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7390201d0d14fb0a61ed7f270ee038c"
I20260810 23:39:23.263214 12921 raft_consensus.cc:3037] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:23.263686 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.243608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:38429 (local address 127.30.33.3:43231)
0810 23:39:23.243809 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.243815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.252668 (+ 8853us) server_negotiation.cc:407] Connection header received
0810 23:39:23.252967 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.252974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.253074 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.253283 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:23.254011 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.255202 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.260216 (+ 5014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.261650 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.261793 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.261941 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.262950 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.263163 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:39:23.264698 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.243210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43231 (local address 127.30.33.2:38429)
0810 23:39:23.252558 (+ 9348us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:23.252577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:23.252794 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:23.253572 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:23.253577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:23.253609 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:23.253795 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.253808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.255581 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.255586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.259923 (+ 4337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.259949 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.262444 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.262452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.262717 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.262760 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:39:23.264202 (+ 1442us) client_negotiation.cc:240] Negotiation successful
0810 23:39:23.264385 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9223,"thread_start_us":11019,"threads_started":1}
I20260810 23:39:23.265944 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ded065e71426ca88b56ea59023aab" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6" is_pre_election: true
I20260810 23:39:23.266258 13446 raft_consensus.cc:2454] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 0.
I20260810 23:39:23.266743 12921 raft_consensus.cc:2454] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 1.
I20260810 23:39:23.267266 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4ded065e71426ca88b56ea59023aab" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6"
I20260810 23:39:23.267510 13446 raft_consensus.cc:3037] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:23.268009 13057 leader_election.cc:302] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, a7390201d0d14fb0a61ed7f270ee038c; no voters:
I20260810 23:39:23.268373 13670 raft_consensus.cc:2781] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:23.271119 13446 raft_consensus.cc:2454] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 1.
I20260810 23:39:23.272766 13670 raft_consensus.cc:686] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 1 LEADER]: Becoming Leader. State: Replica: 669cc74c05394844a74f5a45d70212da, State: Running, Role: LEADER
I20260810 23:39:23.273263 13670 consensus_queue.cc:237] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.277554 12637 catalog_manager.cc:5495] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da reported cstate change: term changed from 0 to 1, leader changed from <none> to 669cc74c05394844a74f5a45d70212da (127.30.33.2). New cstate: current_term: 1 leader_uuid: "669cc74c05394844a74f5a45d70212da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20260810 23:39:23.298259 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.288856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.30.33.2:42105)
0810 23:39:23.289536 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.289543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.289568 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:39:23.289693 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.289698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.289812 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.290657 (+ 845us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:23.290942 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.292329 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.294431 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.295129 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.295829 (+ 700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.296009 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:23.296846 (+ 837us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:23.297699 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.297758 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.297860 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":330,"threads_started":1}
I20260810 23:39:23.303484 13446 raft_consensus.cc:1261] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Refusing update from remote peer 669cc74c05394844a74f5a45d70212da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:23.303498 12921 raft_consensus.cc:1261] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Refusing update from remote peer 669cc74c05394844a74f5a45d70212da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:23.304400 13760 consensus_queue.cc:1035] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:23.305240 13670 consensus_queue.cc:1035] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [LEADER]: Connected to new peer: Peer: permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:23.342029 12637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:23.342329 12637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:23.346235 12637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Sending DeleteTablet for 3 replicas of tablet ad4ded065e71426ca88b56ea59023aab
I20260810 23:39:23.347294 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4ded065e71426ca88b56ea59023aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:23 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:48654
I20260810 23:39:23.347692 13730 tablet_replica.cc:330] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: stopping tablet replica
I20260810 23:39:23.347913 13730 raft_consensus.cc:2229] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:23.348215 13730 raft_consensus.cc:2258] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.349241 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4ded065e71426ca88b56ea59023aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:23 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260810 23:39:23.349572 13731 tablet_replica.cc:330] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: stopping tablet replica
I20260810 23:39:23.349793 13731 raft_consensus.cc:2229] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:23.350049 13731 raft_consensus.cc:2258] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.350919 13141 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4ded065e71426ca88b56ea59023aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:23 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:37632
I20260810 23:39:23.351215 13728 tablet_replica.cc:330] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: stopping tablet replica
I20260810 23:39:23.351424 13728 raft_consensus.cc:2229] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:39:23.352080 13728 raft_consensus.cc:2258] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.353070 13730 ts_tablet_manager.cc:1905] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.354612 13728 ts_tablet_manager.cc:1905] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.360427 12637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:39:23.361706 13730 ts_tablet_manager.cc:1918] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:23.361948 13730 log.cc:1196] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/ad4ded065e71426ca88b56ea59023aab
I20260810 23:39:23.362381 13730 ts_tablet_manager.cc:1939] T ad4ded065e71426ca88b56ea59023aab P a7390201d0d14fb0a61ed7f270ee038c: Deleting consensus metadata
W20260810 23:39:23.362747 12637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:23.364032 12610 catalog_manager.cc:4841] TS a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975): tablet ad4ded065e71426ca88b56ea59023aab (table test-table [id=68c6eb94a8f34b089d9bbd201eb13438]) successfully deleted
I20260810 23:39:23.367257 13728 ts_tablet_manager.cc:1918] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:23.373293 13731 ts_tablet_manager.cc:1905] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.381018 13403 tablet_service.cc:1467] Processing CreateTablet for tablet cf782436ea4346db920c6bff73a96590 (DEFAULT_TABLE table=test-table [id=5fbae3f18e1c4cb9b19e7116ed138343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:23.381562 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf782436ea4346db920c6bff73a96590. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:23.381613 13731 ts_tablet_manager.cc:1918] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:23.381806 13731 log.cc:1196] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/ad4ded065e71426ca88b56ea59023aab
I20260810 23:39:23.382359 13728 log.cc:1196] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/ad4ded065e71426ca88b56ea59023aab
I20260810 23:39:23.382862 13728 ts_tablet_manager.cc:1939] T ad4ded065e71426ca88b56ea59023aab P 669cc74c05394844a74f5a45d70212da: Deleting consensus metadata
I20260810 23:39:23.383723 12888 tablet_service.cc:1467] Processing CreateTablet for tablet cf782436ea4346db920c6bff73a96590 (DEFAULT_TABLE table=test-table [id=5fbae3f18e1c4cb9b19e7116ed138343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:23.378412 13141 tablet_service.cc:1467] Processing CreateTablet for tablet cf782436ea4346db920c6bff73a96590 (DEFAULT_TABLE table=test-table [id=5fbae3f18e1c4cb9b19e7116ed138343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:23.384215 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf782436ea4346db920c6bff73a96590. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:23.384352 12607 catalog_manager.cc:4841] TS 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105): tablet ad4ded065e71426ca88b56ea59023aab (table test-table [id=68c6eb94a8f34b089d9bbd201eb13438]) successfully deleted
I20260810 23:39:23.384488 13141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf782436ea4346db920c6bff73a96590. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:23.390009 13664 tablet_bootstrap.cc:492] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: Bootstrap starting.
I20260810 23:39:23.391094 13662 tablet_bootstrap.cc:492] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: Bootstrap starting.
I20260810 23:39:23.392952 13662 tablet_bootstrap.cc:654] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:23.394058 13665 tablet_bootstrap.cc:492] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: Bootstrap starting.
I20260810 23:39:23.395943 13665 tablet_bootstrap.cc:654] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:23.396378 13731 ts_tablet_manager.cc:1939] T ad4ded065e71426ca88b56ea59023aab P df672b447b394b92a222292262ead3b6: Deleting consensus metadata
I20260810 23:39:23.398509 12610 catalog_manager.cc:4841] TS df672b447b394b92a222292262ead3b6 (127.30.33.3:43231): tablet ad4ded065e71426ca88b56ea59023aab (table test-table [id=68c6eb94a8f34b089d9bbd201eb13438]) successfully deleted
I20260810 23:39:23.398885 13664 tablet_bootstrap.cc:654] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:23.402993 13665 tablet_bootstrap.cc:492] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: No bootstrap required, opened a new log
I20260810 23:39:23.403193 13665 ts_tablet_manager.cc:1397] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 23:39:23.404232 13665 raft_consensus.cc:348] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.404556 13665 raft_consensus.cc:374] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:23.404665 13665 raft_consensus.cc:729] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669cc74c05394844a74f5a45d70212da, State: Initialized, Role: FOLLOWER
I20260810 23:39:23.405068 13665 consensus_queue.cc:260] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.406690 13665 ts_tablet_manager.cc:1428] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:39:23.410799 13662 tablet_bootstrap.cc:492] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: No bootstrap required, opened a new log
I20260810 23:39:23.411063 13662 ts_tablet_manager.cc:1397] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.006s
I20260810 23:39:23.412000 13662 raft_consensus.cc:348] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.412343 13662 raft_consensus.cc:374] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:23.412474 13662 raft_consensus.cc:729] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7390201d0d14fb0a61ed7f270ee038c, State: Initialized, Role: FOLLOWER
I20260810 23:39:23.412806 13662 consensus_queue.cc:260] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.413466 13662 ts_tablet_manager.cc:1428] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:39:23.420295 13664 tablet_bootstrap.cc:492] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: No bootstrap required, opened a new log
I20260810 23:39:23.420502 13664 ts_tablet_manager.cc:1397] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.003s
I20260810 23:39:23.421432 13664 raft_consensus.cc:348] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.421721 13664 raft_consensus.cc:374] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:23.421833 13664 raft_consensus.cc:729] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df672b447b394b92a222292262ead3b6, State: Initialized, Role: FOLLOWER
I20260810 23:39:23.422119 13664 consensus_queue.cc:260] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.425755 13664 ts_tablet_manager.cc:1428] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:39:23.498488 13760 raft_consensus.cc:482] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:23.498819 13760 raft_consensus.cc:504] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.499815 13760 leader_election.cc:288] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975), df672b447b394b92a222292262ead3b6 (127.30.33.3:43231)
I20260810 23:39:23.500582 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf782436ea4346db920c6bff73a96590" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7390201d0d14fb0a61ed7f270ee038c" is_pre_election: true
I20260810 23:39:23.500897 12921 raft_consensus.cc:2454] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 0.
I20260810 23:39:23.501421 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf782436ea4346db920c6bff73a96590" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6" is_pre_election: true
I20260810 23:39:23.501498 13057 leader_election.cc:302] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, a7390201d0d14fb0a61ed7f270ee038c; no voters:
I20260810 23:39:23.502079 13760 raft_consensus.cc:2781] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:23.502210 13760 raft_consensus.cc:482] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:23.502339 13760 raft_consensus.cc:3037] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:23.501720 13446 raft_consensus.cc:2454] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 0.
I20260810 23:39:23.505851 13760 raft_consensus.cc:504] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.506646 13760 leader_election.cc:288] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 election: Requested vote from peers a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975), df672b447b394b92a222292262ead3b6 (127.30.33.3:43231)
I20260810 23:39:23.507357 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf782436ea4346db920c6bff73a96590" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7390201d0d14fb0a61ed7f270ee038c"
I20260810 23:39:23.507377 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf782436ea4346db920c6bff73a96590" candidate_uuid: "669cc74c05394844a74f5a45d70212da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df672b447b394b92a222292262ead3b6"
I20260810 23:39:23.507643 12921 raft_consensus.cc:3037] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:23.507666 13446 raft_consensus.cc:3037] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:23.511050 12921 raft_consensus.cc:2454] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 1.
I20260810 23:39:23.511062 13446 raft_consensus.cc:2454] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669cc74c05394844a74f5a45d70212da in term 1.
I20260810 23:39:23.511677 13057 leader_election.cc:302] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669cc74c05394844a74f5a45d70212da, a7390201d0d14fb0a61ed7f270ee038c; no voters:
I20260810 23:39:23.512107 13760 raft_consensus.cc:2781] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:23.512310 13760 raft_consensus.cc:686] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 1 LEADER]: Becoming Leader. State: Replica: 669cc74c05394844a74f5a45d70212da, State: Running, Role: LEADER
I20260810 23:39:23.512900 13760 consensus_queue.cc:237] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } }
I20260810 23:39:23.521808 12636 catalog_manager.cc:5495] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da reported cstate change: term changed from 0 to 1, leader changed from <none> to 669cc74c05394844a74f5a45d70212da (127.30.33.2). New cstate: current_term: 1 leader_uuid: "669cc74c05394844a74f5a45d70212da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "669cc74c05394844a74f5a45d70212da" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20260810 23:39:23.586766 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.579159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:54477 (local address 127.30.33.1:42975)
0810 23:39:23.579346 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.579352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.580118 (+ 766us) server_negotiation.cc:407] Connection header received
0810 23:39:23.580321 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.580328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.580438 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.580537 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:23.581177 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.582345 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.584496 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.585186 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.585325 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.585910 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:23.586071 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:23.586154 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.586297 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.586389 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:39:23.587090 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.579156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42975 (local address 127.30.33.2:54477)
0810 23:39:23.579985 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:23.580007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:23.580186 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:23.580577 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:23.580582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:23.580594 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:39:23.581002 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.581016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.582523 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.582530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.584320 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.584341 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.585422 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.585429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.585622 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.585697 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:39:23.586175 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:39:23.586186 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:39:23.586272 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:39:23.586439 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":454,"threads_started":1}
I20260810 23:39:23.597070 13446 raft_consensus.cc:1261] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Refusing update from remote peer 669cc74c05394844a74f5a45d70212da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:23.597095 12921 raft_consensus.cc:1261] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Refusing update from remote peer 669cc74c05394844a74f5a45d70212da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:23.597744 13670 consensus_queue.cc:1035] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [LEADER]: Connected to new peer: Peer: permanent_uuid: "df672b447b394b92a222292262ead3b6" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:23.598280 13670 consensus_queue.cc:1035] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7390201d0d14fb0a61ed7f270ee038c" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:23.643575 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.637391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42105 (local address 127.30.33.1:60681)
0810 23:39:23.637994 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:23.638020 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:23.638172 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:23.638515 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:23.638519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:23.638531 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:39:23.638754 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.638763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.639886 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.639892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.641397 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:23.641412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.642482 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.642489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:23.642618 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.642659 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:39:23.643069 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:39:23.643076 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:39:23.643126 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 23:39:23.643256 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":308,"threads_started":1}
I20260810 23:39:23.643601 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.637190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:60681 (local address 127.30.33.2:42105)
0810 23:39:23.637347 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.637352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.638122 (+ 770us) server_negotiation.cc:407] Connection header received
0810 23:39:23.638315 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.638321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.638413 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.638499 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:23.638888 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.639751 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.641571 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.642321 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.642458 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.642836 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:23.642975 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:23.643066 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.643172 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.643265 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:39:23.689806 12637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:23.690140 12637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:23.693737 12637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81353e539f0948b0b35dbc74b8307117: Sending DeleteTablet for 3 replicas of tablet cf782436ea4346db920c6bff73a96590
I20260810 23:39:23.694818 13141 tablet_service.cc:1514] Processing DeleteTablet for tablet cf782436ea4346db920c6bff73a96590 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:23 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:37632
I20260810 23:39:23.695230 13728 tablet_replica.cc:330] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: stopping tablet replica
I20260810 23:39:23.695474 13728 raft_consensus.cc:2229] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:39:23.695890 13728 raft_consensus.cc:2258] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.696915 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 12766
I20260810 23:39:23.697665 13728 ts_tablet_manager.cc:1905] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.699375 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet cf782436ea4346db920c6bff73a96590 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:23 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260810 23:39:23.699728 13731 tablet_replica.cc:330] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: stopping tablet replica
I20260810 23:39:23.699922 13731 raft_consensus.cc:2229] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:23.700198 13731 raft_consensus.cc:2258] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.701543 13731 ts_tablet_manager.cc:1905] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:23.705408 13728 ts_tablet_manager.cc:1918] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:39:23.705590 13728 log.cc:1196] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/cf782436ea4346db920c6bff73a96590
I20260810 23:39:23.706089 13728 ts_tablet_manager.cc:1939] T cf782436ea4346db920c6bff73a96590 P 669cc74c05394844a74f5a45d70212da: Deleting consensus metadata
I20260810 23:39:23.707540 12607 catalog_manager.cc:4841] TS 669cc74c05394844a74f5a45d70212da (127.30.33.2:42105): tablet cf782436ea4346db920c6bff73a96590 (table test-table [id=5fbae3f18e1c4cb9b19e7116ed138343]) successfully deleted
I20260810 23:39:23.707976 13731 ts_tablet_manager.cc:1918] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:39:23.707966 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet cf782436ea4346db920c6bff73a96590 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:23 UTC) from {username='oryx', principal='oryx/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:48654
I20260810 23:39:23.708105 13731 log.cc:1196] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/cf782436ea4346db920c6bff73a96590
I20260810 23:39:23.708324 13730 tablet_replica.cc:330] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: stopping tablet replica
I20260810 23:39:23.708516 13730 raft_consensus.cc:2229] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:23.708562 13731 ts_tablet_manager.cc:1939] T cf782436ea4346db920c6bff73a96590 P df672b447b394b92a222292262ead3b6: Deleting consensus metadata
I20260810 23:39:23.708802 13730 raft_consensus.cc:2258] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:23.709645 12610 catalog_manager.cc:4841] TS df672b447b394b92a222292262ead3b6 (127.30.33.3:43231): tablet cf782436ea4346db920c6bff73a96590 (table test-table [id=5fbae3f18e1c4cb9b19e7116ed138343]) successfully deleted
I20260810 23:39:23.710282 13730 ts_tablet_manager.cc:1905] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:39:23 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.1@KRBTEST.COM
I20260810 23:39:23.717584 13730 ts_tablet_manager.cc:1918] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:39:23.717753 13730 log.cc:1196] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/cf782436ea4346db920c6bff73a96590
I20260810 23:39:23.718262 13730 ts_tablet_manager.cc:1939] T cf782436ea4346db920c6bff73a96590 P a7390201d0d14fb0a61ed7f270ee038c: Deleting consensus metadata
I20260810 23:39:23.719657 12610 catalog_manager.cc:4841] TS a7390201d0d14fb0a61ed7f270ee038c (127.30.33.1:42975): tablet cf782436ea4346db920c6bff73a96590 (table test-table [id=5fbae3f18e1c4cb9b19e7116ed138343]) successfully deleted
I20260810 23:39:23.725867 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.698356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.30.33.1:42975)
0810 23:39:23.698556 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.698562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.698580 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:39:23.698640 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.698643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.698744 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.698838 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:23.700933 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.701927 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.704091 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.704761 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.704867 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.718306 (+ 13439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:23.718329 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:23.718350 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:23.718398 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:23.722447 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:23.723444 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:23.723480 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:23.723485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:23.723645 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:23.724112 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:23.724117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:23.724121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:23.724549 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:23.724857 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.725300 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.725535 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:39:23.726775 13006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:23.884902 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 12766
I20260810 23:39:23.925609 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 13021
Aug 10 23:39:23 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.2@KRBTEST.COM
I20260810 23:39:23.941709 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:23.926398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.30.33.2:42105)
0810 23:39:23.926989 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:23.926994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:23.927009 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:39:23.927075 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:23.927078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:23.927139 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:23.927238 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:23.928541 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.929266 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.930549 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:23.931128 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:23.931246 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:23.935427 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:23.935451 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:23.935532 (+ 81us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:23.935584 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:23.938478 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:23.939248 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:23.939259 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:23.939263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:23.939418 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:23.939990 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:23.939996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:23.940000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:23.940423 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:23.940751 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:23.941128 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 23:39:23.941373 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475}
I20260810 23:39:23.942417 13252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:24.070740 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 13021
I20260810 23:39:24.101992 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 13267
Aug 10 23:39:24 dist-test-slave-pww0 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.33.3@KRBTEST.COM
I20260810 23:39:24.118198 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:24.102819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.30.33.3:43231)
0810 23:39:24.103357 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:24.103362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:24.103380 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:39:24.103436 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:24.103440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:24.103539 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:24.103632 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:24.104966 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:24.105750 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:24.106990 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:24.107403 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:24.107582 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:24.112061 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:24.112079 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:24.112097 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:24.112141 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:24.115038 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:24.115954 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:24.115963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:24.115965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:24.116126 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:24.116591 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:24.116595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:24.116598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:24.117007 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:24.117241 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:24.117634 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 23:39:24.117841 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":303,"threads_started":1}
I20260810 23:39:24.118875 13528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:24.243516 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 13267
I20260810 23:39:24.276333 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 12573
I20260810 23:39:24.277683 12709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:24.399125 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 12573
2026-08-10T23:39:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6566 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:39:24 dist-test-slave-pww0 krb5kdc[13809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:39:24 dist-test-slave-pww0 krb5kdc[13809](info): set up 2 sockets
Aug 10 23:39:24 dist-test-slave-pww0 krb5kdc[13809](info): commencing operation
krb5kdc: starting...
W20260810 23:39:25.706321 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.009s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:39:25 dist-test-slave-pww0 krb5kdc[13809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:39:26.003592 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:44049
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:44049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:26.357329 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:26.357669 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:26.357806 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:26.365383 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:26.365526 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:26.365589 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:26.365671 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:26.365741 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:26.376480 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:44049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:26.377940 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:26.379352 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:26.391916 13854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:26.391861 13855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:26.391826 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:26.392843 13841 server_base.cc:1034] running on GCE node
I20260810 23:39:26.393707 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:26.395352 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:26.396584 13841 hybrid_clock.cc:648] HybridClock initialized: now 1786405166396521 us; error 53 us; skew 500 ppm
I20260810 23:39:26.396986 13841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T23:39:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2060 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:39:26 dist-test-slave-pww0 krb5kdc[13873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:39:26 dist-test-slave-pww0 krb5kdc[13873](info): set up 2 sockets
Aug 10 23:39:26 dist-test-slave-pww0 krb5kdc[13873](info): commencing operation
krb5kdc: starting...
W20260810 23:39:27.758952 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:39:27 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:39:28.063122 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41311
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:28.428696 13893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:28.429033 13893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:28.429169 13893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:28.436867 13893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:28.437000 13893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:28.437067 13893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:28.437151 13893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:28.437217 13893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:28.447988 13893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:28.449520 13893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:28.450937 13893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:28.463910 13903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:28.463888 13904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:28.463827 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:28.465034 13893 server_base.cc:1034] running on GCE node
I20260810 23:39:28.465975 13893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:28.467638 13893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:28.468884 13893 hybrid_clock.cc:648] HybridClock initialized: now 1786405168468826 us; error 44 us; skew 500 ppm
I20260810 23:39:28.469270 13893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:28 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405168, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:28.472716 13893 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:39:28.474406 13893 webserver.cc:460] Webserver started at http://127.30.33.62:36727/ using document root <none> and password file <none>
I20260810 23:39:28.475140 13893 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:28.475265 13893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:28.475634 13893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:28.478772 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "812a84bfd3ef40148a5b62cdddc7c2b3"
format_stamp: "Formatted at 2026-08-10 23:39:28 on dist-test-slave-pww0"
server_key: "3810dd1bc638beeb15b64bb276bd2f5f"
server_key_iv: "c2d3031c59ba9e7349defce6929fc8e7"
server_key_version: "encryptionkey@0"
I20260810 23:39:28.479612 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "812a84bfd3ef40148a5b62cdddc7c2b3"
format_stamp: "Formatted at 2026-08-10 23:39:28 on dist-test-slave-pww0"
server_key: "3810dd1bc638beeb15b64bb276bd2f5f"
server_key_iv: "c2d3031c59ba9e7349defce6929fc8e7"
server_key_version: "encryptionkey@0"
I20260810 23:39:28.485162 13893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:39:28.488852 13919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:28.490334 13893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:39:28.490527 13893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "812a84bfd3ef40148a5b62cdddc7c2b3"
format_stamp: "Formatted at 2026-08-10 23:39:28 on dist-test-slave-pww0"
server_key: "3810dd1bc638beeb15b64bb276bd2f5f"
server_key_iv: "c2d3031c59ba9e7349defce6929fc8e7"
server_key_version: "encryptionkey@0"
I20260810 23:39:28.490737 13893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:28.508078 13893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:28.512809 13893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:28.513099 13893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:28.531940 13893 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:41311
I20260810 23:39:28.531981 14008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:41311 every 8 connection(s)
I20260810 23:39:28.533483 13893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:28.537408 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:28.543814 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 13893
I20260810 23:39:28.544025 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:28.544663 30852 external_mini_cluster.cc:1428] Setting key 123af731ec1294c13f9c61985c970575
I20260810 23:39:28.548734 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Bootstrap starting.
I20260810 23:39:28.552695 14010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:28.553866 14010 log.cc:824] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:28.557369 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: No bootstrap required, opened a new log
Aug 10 23:39:28 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:28.566789 14010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:28.567203 14010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:28.567361 14010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812a84bfd3ef40148a5b62cdddc7c2b3, State: Initialized, Role: FOLLOWER
I20260810 23:39:28.568274 14010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:28.568617 14010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:28.568828 14010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:28.569034 14010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:28.570320 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:28.546863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.30.33.62:41311)
0810 23:39:28.547707 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:28.547720 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:28.547771 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:39:28.549645 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:28.549672 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:28.550105 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:28.550493 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:28.552531 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:28.553989 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:28.555638 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:28.556350 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:28.556497 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:28.561509 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:28.561518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:28.561537 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:28.561589 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:28.564964 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:28.565913 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:28.565921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:28.565929 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:28.566113 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:28.566527 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:28.566530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:28.566532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:28.567140 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:28.567517 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:28.568675 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0810 23:39:28.569129 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":346,"threads_started":1}
I20260810 23:39:28.571645 14010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:28.572330 14010 leader_election.cc:302] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 812a84bfd3ef40148a5b62cdddc7c2b3; no voters:
I20260810 23:39:28.573007 14010 leader_election.cc:288] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:28.573129 14019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:28.574597 14019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 LEADER]: Becoming Leader. State: Replica: 812a84bfd3ef40148a5b62cdddc7c2b3, State: Running, Role: LEADER
I20260810 23:39:28.575098 14010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:28.575150 14019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:28.577936 14021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812a84bfd3ef40148a5b62cdddc7c2b3. Latest consensus state: current_term: 1 leader_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } } }
I20260810 23:39:28.578378 14021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:28.579420 14020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } } }
I20260810 23:39:28.580163 14020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:28.580852 14033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:28.586508 14033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:28.596036 14033 catalog_manager.cc:1347] Generated new cluster ID: e72e6e63482a41d7a5789a904358db73
I20260810 23:39:28.596174 14033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:28.607344 14033 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:28.608771 14033 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:28.622045 14033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Generated new TSK 0
I20260810 23:39:28.622977 14033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:39:28.698486 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41311
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:29.054214 14052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:29.054534 14052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:29.054666 14052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:29.062422 14052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:29.062564 14052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:29.062721 14052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:29.073150 14052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:29.074558 14052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:29.076166 14052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:29.090794 14064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:29.090819 14061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:29.091398 14066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:29.091856 14052 server_base.cc:1034] running on GCE node
I20260810 23:39:29.092545 14052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:29.093621 14052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:29.094878 14052 hybrid_clock.cc:648] HybridClock initialized: now 1786405169094801 us; error 59 us; skew 500 ppm
I20260810 23:39:29.095268 14052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:29 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405169, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:29.103036 14052 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:39:29.104875 14052 webserver.cc:460] Webserver started at http://127.30.33.1:37823/ using document root <none> and password file <none>
I20260810 23:39:29.105666 14052 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:29.105799 14052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:29.106112 14052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:29.109302 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "4da4ea0e6c8f440e9698511ad0f1b914"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "61c22653e3a242e12f0df449dd670559"
server_key_iv: "b565094821f0b8be51dd003c75d71893"
server_key_version: "encryptionkey@0"
I20260810 23:39:29.110222 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "4da4ea0e6c8f440e9698511ad0f1b914"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "61c22653e3a242e12f0df449dd670559"
server_key_iv: "b565094821f0b8be51dd003c75d71893"
server_key_version: "encryptionkey@0"
I20260810 23:39:29.116164 14052 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:39:29.120056 14081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:29.121688 14052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:39:29.121887 14052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "4da4ea0e6c8f440e9698511ad0f1b914"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "61c22653e3a242e12f0df449dd670559"
server_key_iv: "b565094821f0b8be51dd003c75d71893"
server_key_version: "encryptionkey@0"
I20260810 23:39:29.122109 14052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:29.151423 14052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:29.156090 14052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:29.156390 14052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:29.157450 14052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:29.159027 14052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:29.159111 14052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:29.159215 14052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:29.159277 14052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:29.192571 14052 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43383
I20260810 23:39:29.192628 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43383 every 8 connection(s)
I20260810 23:39:29.194007 14052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:29.199558 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 14052
I20260810 23:39:29.199805 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:39:29.200634 30852 external_mini_cluster.cc:1428] Setting key 4be80c79c98868cb0527de63f74d2f73
Aug 10 23:39:29 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405169, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:29.217339 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:29.196974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:45563 (local address 127.30.33.62:41311)
0810 23:39:29.197442 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:29.197447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:29.198558 (+ 1111us) server_negotiation.cc:407] Connection header received
0810 23:39:29.201236 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:29.201242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:29.201337 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:29.201475 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:29.203438 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.204242 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.206471 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.206886 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.206985 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:29.211219 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:29.211224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:29.211227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:29.211264 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:29.213892 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:29.214808 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:29.214811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:29.214814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:29.214953 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:29.215437 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:29.215441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:29.215444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:29.215770 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:29.215886 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:29.216797 (+ 911us) server_negotiation.cc:299] Negotiation successful
0810 23:39:29.216982 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
I20260810 23:39:29.218433 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:29.197590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:45563)
0810 23:39:29.198362 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:29.198409 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:29.200865 (+ 2456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:29.201759 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:29.201772 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:29.202385 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:29.203236 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:29.203268 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.204420 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.204425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:29.206306 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:29.206323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.207115 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.207122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:29.207395 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:29.208153 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:29.208181 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:29.210943 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:29.214053 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:29.214062 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:29.214081 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:29.214691 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:29.215056 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:29.215061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:29.215064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:29.215330 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:29.215904 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:29.215913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:29.216381 (+ 468us) client_negotiation.cc:769] Sending connection context
0810 23:39:29.216720 (+ 339us) client_negotiation.cc:240] Negotiation successful
0810 23:39:29.217263 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":285,"threads_started":1}
I20260810 23:39:29.220151 14296 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41311
I20260810 23:39:29.220501 14296 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:29.221292 14296 heartbeater.cc:511] Master 127.30.33.62:41311 requested a full tablet report, sending...
I20260810 23:39:29.223579 13943 ts_manager.cc:194] Registered new tserver with Master: 4da4ea0e6c8f440e9698511ad0f1b914 (127.30.33.1:43383)
I20260810 23:39:29.225211 13943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:45563
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:39:29.266187 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41311
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:29.622390 14306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:29.622843 14306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:29.622999 14306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:29.630818 14306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:29.630975 14306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:29.631145 14306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:29.641482 14306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:29.642926 14306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:29.644402 14306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:29.658624 14316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:29.658661 14315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:29.658643 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:29.659351 14306 server_base.cc:1034] running on GCE node
I20260810 23:39:29.660121 14306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:29.661249 14306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:29.662472 14306 hybrid_clock.cc:648] HybridClock initialized: now 1786405169662418 us; error 53 us; skew 500 ppm
I20260810 23:39:29.662822 14306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:29 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405169, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:29.666312 14306 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:39:29.667963 14306 webserver.cc:460] Webserver started at http://127.30.33.2:42635/ using document root <none> and password file <none>
I20260810 23:39:29.668709 14306 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:29.668843 14306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:29.669164 14306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:29.672406 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "4319c7828bb849a8b707d1f35f9d58e6"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "2bccc89877cae075a1efa480273a520f"
server_key_iv: "d7a83bf98debe9e8c823b806aea3296a"
server_key_version: "encryptionkey@0"
I20260810 23:39:29.673250 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "4319c7828bb849a8b707d1f35f9d58e6"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "2bccc89877cae075a1efa480273a520f"
server_key_iv: "d7a83bf98debe9e8c823b806aea3296a"
server_key_version: "encryptionkey@0"
I20260810 23:39:29.679190 14306 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:39:29.683009 14329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:29.684496 14306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:29.684670 14306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "4319c7828bb849a8b707d1f35f9d58e6"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "2bccc89877cae075a1efa480273a520f"
server_key_iv: "d7a83bf98debe9e8c823b806aea3296a"
server_key_version: "encryptionkey@0"
I20260810 23:39:29.684877 14306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:29.703042 14306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:29.707321 14306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:29.707630 14306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:29.708710 14306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:29.710162 14306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:29.710268 14306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:29.710403 14306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:29.710457 14306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:29.742993 14306 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:45789
I20260810 23:39:29.743033 14544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:45789 every 8 connection(s)
I20260810 23:39:29.744462 14306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:29.746910 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 14306
I20260810 23:39:29.747089 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:39:29.747701 30852 external_mini_cluster.cc:1428] Setting key 01e6e2b25de0ca5f8bc58eaa0d107825
Aug 10 23:39:29 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405169, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:29.768568 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:29.747266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:34741 (local address 127.30.33.62:41311)
0810 23:39:29.747780 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:29.747786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:29.748956 (+ 1170us) server_negotiation.cc:407] Connection header received
0810 23:39:29.751451 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:29.751483 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:29.751571 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:29.751661 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:29.754013 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.754790 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.757147 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.757589 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.757719 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:29.762164 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:29.762169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:29.762173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:29.762215 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:29.764924 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:29.765934 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:29.765938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:29.765941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:29.766080 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:29.766668 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:29.766670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:29.766672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:29.766974 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:29.767092 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:29.768075 (+ 983us) server_negotiation.cc:299] Negotiation successful
0810 23:39:29.768288 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260810 23:39:29.770076 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:29.747916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:34741)
0810 23:39:29.748786 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:29.748844 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:29.751085 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:29.752003 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:29.752016 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:29.752874 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:29.753765 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:29.753797 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.754996 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.755001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:29.756948 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:29.756969 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:29.757858 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:29.757865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:29.758185 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:29.759002 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:29.759039 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:29.761877 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:29.765121 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:29.765132 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:29.765154 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:29.765808 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:29.766213 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:29.766218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:29.766222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:29.766562 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:29.767103 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:29.767111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:29.767637 (+ 526us) client_negotiation.cc:769] Sending connection context
0810 23:39:29.767950 (+ 313us) client_negotiation.cc:240] Negotiation successful
0810 23:39:29.768606 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":338,"threads_started":1}
I20260810 23:39:29.771816 14545 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41311
I20260810 23:39:29.772140 14545 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:29.772939 14545 heartbeater.cc:511] Master 127.30.33.62:41311 requested a full tablet report, sending...
I20260810 23:39:29.774657 13943 ts_manager.cc:194] Registered new tserver with Master: 4319c7828bb849a8b707d1f35f9d58e6 (127.30.33.2:45789)
I20260810 23:39:29.775686 13943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:34741
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:39:29.817011 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41311
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:30.166476 14554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:30.166853 14554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:30.167003 14554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:30.175374 14554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:30.175565 14554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:30.175731 14554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:30.186301 14554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:30.187767 14554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:30.189213 14554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:30.203703 14562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:30.203657 14566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:30.203598 14564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:30.204535 14554 server_base.cc:1034] running on GCE node
I20260810 23:39:30.205207 14554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:30.206298 14554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:30.207535 14554 hybrid_clock.cc:648] HybridClock initialized: now 1786405170207460 us; error 72 us; skew 500 ppm
I20260810 23:39:30.207880 14554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:30 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405170, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:30.211128 14554 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:39:30.212774 14554 webserver.cc:460] Webserver started at http://127.30.33.3:37807/ using document root <none> and password file <none>
I20260810 23:39:30.213464 14554 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:30.213567 14554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:30.213853 14554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:30.217073 14554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "4c30945f1b7841838b5237ca15379a49"
format_stamp: "Formatted at 2026-08-10 23:39:30 on dist-test-slave-pww0"
server_key: "0651371e112e65e28b43a521f35c1fcb"
server_key_iv: "5b087e95c86f6f388627444f06bdd6bf"
server_key_version: "encryptionkey@0"
I20260810 23:39:30.217881 14554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "4c30945f1b7841838b5237ca15379a49"
format_stamp: "Formatted at 2026-08-10 23:39:30 on dist-test-slave-pww0"
server_key: "0651371e112e65e28b43a521f35c1fcb"
server_key_iv: "5b087e95c86f6f388627444f06bdd6bf"
server_key_version: "encryptionkey@0"
I20260810 23:39:30.223680 14554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:39:30.227591 14579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:30.228767 14296 heartbeater.cc:503] Master 127.30.33.62:41311 was elected leader, sending a full tablet report...
I20260810 23:39:30.229128 14554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:39:30.229296 14554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "4c30945f1b7841838b5237ca15379a49"
format_stamp: "Formatted at 2026-08-10 23:39:30 on dist-test-slave-pww0"
server_key: "0651371e112e65e28b43a521f35c1fcb"
server_key_iv: "5b087e95c86f6f388627444f06bdd6bf"
server_key_version: "encryptionkey@0"
I20260810 23:39:30.229501 14554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:30.249928 14554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:30.254890 14554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:30.255220 14554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:30.256322 14554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:30.257809 14554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:30.257893 14554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:30.258024 14554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:30.258080 14554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:30.290421 14554 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:36041
I20260810 23:39:30.290470 14797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:36041 every 8 connection(s)
I20260810 23:39:30.291801 14554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:30.297696 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 14554
I20260810 23:39:30.297895 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:39:30.298364 30852 external_mini_cluster.cc:1428] Setting key 2c7b1d343b044fc8a1698f0bd97635e1
Aug 10 23:39:30 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405170, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:30.312783 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:30.294444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:54381 (local address 127.30.33.62:41311)
0810 23:39:30.294940 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:30.294946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:30.296256 (+ 1310us) server_negotiation.cc:407] Connection header received
0810 23:39:30.298207 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:30.298213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:30.298313 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:30.298416 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:30.300158 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.300980 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.302584 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.303002 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.303128 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:30.306811 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:30.306817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:30.306820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:30.306859 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:30.309438 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.310267 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.310272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.310274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.310411 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.310866 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.310870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.310872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.311153 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:30.311317 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:30.312142 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 23:39:30.312400 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":211,"threads_started":1}
I20260810 23:39:30.313886 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:30.295149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:54381)
0810 23:39:30.296071 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:30.296125 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:30.297872 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:30.298708 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:30.298717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:30.299289 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:30.299976 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:30.299994 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.301135 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.301139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:30.302421 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:30.302435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.303251 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.303259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:30.303558 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:30.304212 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:30.304245 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:30.306544 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:30.309606 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:30.309616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:30.309636 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:30.310101 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:30.310526 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:30.310531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:30.310535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:30.310746 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:30.311295 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:30.311304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:30.311739 (+ 435us) client_negotiation.cc:769] Sending connection context
0810 23:39:30.312005 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 23:39:30.312583 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":402,"threads_started":1}
I20260810 23:39:30.315236 14798 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41311
I20260810 23:39:30.315623 14798 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:30.316406 14798 heartbeater.cc:511] Master 127.30.33.62:41311 requested a full tablet report, sending...
I20260810 23:39:30.317909 13943 ts_manager.cc:194] Registered new tserver with Master: 4c30945f1b7841838b5237ca15379a49 (127.30.33.3:36041)
I20260810 23:39:30.318743 13943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:54381
I20260810 23:39:30.326779 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab.
Entry for principal oryx/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab.
WARNING: no policy specified for oryx/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab.
Entry for principal oryx/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab.
WARNING: no policy specified for oryx/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab.
Entry for principal oryx/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab.
I20260810 23:39:30.510991 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 14052
Aug 10 23:39:30 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:39:30.529104 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:30.512391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.30.33.1:43383)
0810 23:39:30.513190 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:30.513197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:30.513228 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:39:30.513299 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:30.513306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:30.513527 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:30.513688 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:30.515148 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.516405 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.517683 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.518245 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.518343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:30.522909 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:30.522928 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:30.522945 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:30.522992 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:30.525942 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.526778 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.526787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.526789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.526939 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.527369 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.527373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.527376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.527890 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:30.528145 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:30.528467 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 23:39:30.528693 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":426,"threads_started":1}
I20260810 23:39:30.530481 14291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:30.617209 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 14052
I20260810 23:39:30.652969 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 14306
Aug 10 23:39:30 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:39:30.671204 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:30.653955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.30.33.2:45789)
0810 23:39:30.654586 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:30.654593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:30.654621 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:39:30.654717 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:30.654724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:30.654938 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:30.655154 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:30.656570 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.657651 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.659121 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.659836 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.659948 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:30.664804 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:30.664824 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:30.664841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:30.664886 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:30.668057 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.668992 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.669002 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.669004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.669157 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.669640 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.669644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.669646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.670075 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:30.670356 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:30.670640 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 23:39:30.670891 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":329,"threads_started":1}
I20260810 23:39:30.672701 14536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:30.757030 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 14306
I20260810 23:39:30.781891 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 14554
Aug 10 23:39:30 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:39:30.799324 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:30.782940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.30.33.3:36041)
0810 23:39:30.783531 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:30.783538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:30.783572 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:39:30.783646 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:30.783653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:30.783881 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:30.784048 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:30.785447 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.786457 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.787942 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:30.788531 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:30.788654 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:30.793105 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:30.793121 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:30.793138 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:30.793183 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:30.796281 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.797061 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.797069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.797072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.797212 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:30.797603 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:30.797606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:30.797608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:30.797980 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:30.798222 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:30.798748 (+ 526us) server_negotiation.cc:299] Negotiation successful
0810 23:39:30.798987 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":244,"threads_started":1}
I20260810 23:39:30.800670 14788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:30.884898 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 14554
I20260810 23:39:30.906425 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 13893
I20260810 23:39:30.907727 13999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:31.016072 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 13893
I20260810 23:39:31.044610 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41311
--webserver_interface=127.30.33.62
--webserver_port=36727
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:31.407764 14821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:31.408087 14821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:31.408227 14821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:31.415961 14821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:31.416117 14821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:31.416195 14821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:31.416291 14821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:31.416350 14821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:31.426837 14821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=36727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:31.428308 14821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:31.429694 14821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:31.441970 14830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:31.442036 14829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:31.442005 14834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:31.443037 14821 server_base.cc:1034] running on GCE node
I20260810 23:39:31.443917 14821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:31.445466 14821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:31.446702 14821 hybrid_clock.cc:648] HybridClock initialized: now 1786405171446636 us; error 53 us; skew 500 ppm
I20260810 23:39:31.447069 14821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:31 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405171, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:31.450356 14821 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:39:31.452032 14821 webserver.cc:460] Webserver started at http://127.30.33.62:36727/ using document root <none> and password file <none>
I20260810 23:39:31.452777 14821 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:31.452909 14821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:31.458979 14821 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:39:31.462096 14846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:31.463588 14821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:31.463778 14821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "812a84bfd3ef40148a5b62cdddc7c2b3"
format_stamp: "Formatted at 2026-08-10 23:39:28 on dist-test-slave-pww0"
server_key: "3810dd1bc638beeb15b64bb276bd2f5f"
server_key_iv: "c2d3031c59ba9e7349defce6929fc8e7"
server_key_version: "encryptionkey@0"
I20260810 23:39:31.464913 14821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:31.484364 14821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:31.489059 14821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:31.489337 14821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:31.508141 14821 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:41311
I20260810 23:39:31.508167 14941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:41311 every 8 connection(s)
I20260810 23:39:31.509770 14821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:31.515970 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 14821
I20260810 23:39:31.516781 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:43383
--local_ip_for_outbound_sockets=127.30.33.1
--tserver_master_addrs=127.30.33.62:41311
--webserver_port=37823
--webserver_interface=127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:39:31.517174 14946 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:39:31.520763 14946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Bootstrap starting.
I20260810 23:39:31.544334 14946 log.cc:824] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:31.557471 14946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:39:31.558205 14946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Bootstrap complete.
I20260810 23:39:31.572305 14946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:31.573616 14946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812a84bfd3ef40148a5b62cdddc7c2b3, State: Initialized, Role: FOLLOWER
I20260810 23:39:31.574419 14946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:31.574728 14946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:31.574870 14946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:31.575065 14946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:39:31.580855 14946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:31.581527 14946 leader_election.cc:302] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 812a84bfd3ef40148a5b62cdddc7c2b3; no voters:
I20260810 23:39:31.582242 14946 leader_election.cc:288] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:39:31.582459 14952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:39:31.583534 14952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [term 2 LEADER]: Becoming Leader. State: Replica: 812a84bfd3ef40148a5b62cdddc7c2b3, State: Running, Role: LEADER
I20260810 23:39:31.584160 14952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } }
I20260810 23:39:31.584627 14946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:31.587652 14956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812a84bfd3ef40148a5b62cdddc7c2b3. Latest consensus state: current_term: 2 leader_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } } }
I20260810 23:39:31.587531 14955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812a84bfd3ef40148a5b62cdddc7c2b3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41311 } } }
I20260810 23:39:31.588454 14956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:31.588743 14955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:31.589689 14963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:31.603451 14963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:31.605495 14963 catalog_manager.cc:1259] Loaded cluster ID: e72e6e63482a41d7a5789a904358db73
I20260810 23:39:31.605607 14963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:31.609529 14963 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:31.611155 14963 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 812a84bfd3ef40148a5b62cdddc7c2b3: Loaded TSK: 0
I20260810 23:39:31.612839 14963 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:39:31.906196 14950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:31.906548 14950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:31.906688 14950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:31.914700 14950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:31.914871 14950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:31.915030 14950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:31.925758 14950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.1.keytab
--principal=oryx/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:43383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=37823
--tserver_master_addrs=127.30.33.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:31.927243 14950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:31.929001 14950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:31.945299 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:31.945327 14990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:31.946213 14996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:31.946646 14950 server_base.cc:1034] running on GCE node
I20260810 23:39:31.947512 14950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:31.948848 14950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:31.950106 14950 hybrid_clock.cc:648] HybridClock initialized: now 1786405171950039 us; error 49 us; skew 500 ppm
I20260810 23:39:31.950583 14950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:31 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405171, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:31.954994 14950 init.cc:377] Logged in from keytab as oryx/127.30.33.1@KRBTEST.COM (short username oryx)
I20260810 23:39:31.957226 14950 webserver.cc:460] Webserver started at http://127.30.33.1:37823/ using document root <none> and password file <none>
I20260810 23:39:31.958235 14950 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:31.958402 14950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:31.965901 14950 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:39:31.969666 15009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:31.971354 14950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:39:31.971671 14950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "4da4ea0e6c8f440e9698511ad0f1b914"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "61c22653e3a242e12f0df449dd670559"
server_key_iv: "b565094821f0b8be51dd003c75d71893"
server_key_version: "encryptionkey@0"
I20260810 23:39:31.972877 14950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:31.988955 14950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:31.993446 14950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:31.993705 14950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:31.994721 14950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:31.996222 14950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:31.996315 14950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:31.996402 14950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:31.996456 14950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:32.028968 14950 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43383
I20260810 23:39:32.029029 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43383 every 8 connection(s)
I20260810 23:39:32.030573 14950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:32.034157 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 14950
I20260810 23:39:32.035029 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:45789
--local_ip_for_outbound_sockets=127.30.33.2
--tserver_master_addrs=127.30.33.62:41311
--webserver_port=42635
--webserver_interface=127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:32.051925 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:32.033807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:42495)
0810 23:39:32.034705 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:32.034755 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:32.036817 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:32.038750 (+ 1933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:32.038759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:32.039341 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:32.040019 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.040044 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.042497 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.042502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.043948 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.043965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.045436 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.045442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.045647 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.046406 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:32.046431 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:32.050395 (+ 3964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
W20260810 23:39:32.052107 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:32.033869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:42495 (local address 127.30.33.62:41311)
0810 23:39:32.036033 (+ 2164us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:32.036050 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:32.036108 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:39:32.037541 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:32.037562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:32.038095 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:32.038571 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:32.040262 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.042265 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.044127 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.045197 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.045403 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.050881 (+ 5478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:42495: BlockingRecv error: recv got EOF from 127.30.33.1:42495 (error 108)
Metrics: {"server-negotiator.queue_time_us":1887,"thread_start_us":1661,"threads_started":1}
W20260810 23:39:32.052990 15230 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:41311 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.33.62:41311: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:32.062655 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:32.053675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:59879)
0810 23:39:32.053903 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:32.053924 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:32.054039 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:32.054366 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:32.054369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:32.054709 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:32.054864 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.054873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.056329 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.056334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.057278 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.057294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.058192 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.058197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.058364 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.059011 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:32.059030 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:32.062409 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 23:39:32.062996 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:32.053874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:59879 (local address 127.30.33.62:41311)
0810 23:39:32.054063 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:32.054068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:32.054089 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:39:32.054162 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:32.054167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:32.054241 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:32.054333 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:32.055029 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.056137 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.057446 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.058011 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.058178 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.062746 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:59879: BlockingRecv error: recv got EOF from 127.30.33.1:59879 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:32.072814 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:32.063438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:60925)
0810 23:39:32.063679 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:32.063701 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:32.063833 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:32.064177 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:32.064183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:32.064597 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:32.064830 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.064844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.065929 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.065935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.067497 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.067519 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.068310 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.068317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.068538 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.069166 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:32.069191 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:32.072576 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 23:39:32.073165 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:32.063679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:60925 (local address 127.30.33.62:41311)
0810 23:39:32.063828 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:32.063833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:32.063851 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:39:32.063949 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:32.063953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:32.064028 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:32.064109 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:32.064981 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.065769 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.067668 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.068131 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.068249 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.072888 (+ 4639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:60925: BlockingRecv error: recv got EOF from 127.30.33.1:60925 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 23:39:32.073175 15230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:39:32.396414 15236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:32.396782 15236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:32.396937 15236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:32.404567 15236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:32.404726 15236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:32.404887 15236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:32.415143 15236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.2.keytab
--principal=oryx/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:45789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=42635
--tserver_master_addrs=127.30.33.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:32.416606 15236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:32.418005 15236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:32.432024 15244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:32.431986 15249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:32.432060 15246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:32.433324 15236 server_base.cc:1034] running on GCE node
I20260810 23:39:32.433976 15236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:32.435000 15236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:32.436196 15236 hybrid_clock.cc:648] HybridClock initialized: now 1786405172436131 us; error 56 us; skew 500 ppm
I20260810 23:39:32.436504 15236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405172, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:32.440006 15236 init.cc:377] Logged in from keytab as oryx/127.30.33.2@KRBTEST.COM (short username oryx)
I20260810 23:39:32.441684 15236 webserver.cc:460] Webserver started at http://127.30.33.2:42635/ using document root <none> and password file <none>
I20260810 23:39:32.442440 15236 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:32.442574 15236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:32.448549 15236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:39:32.451689 15263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:32.453231 15236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:39:32.453401 15236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "4319c7828bb849a8b707d1f35f9d58e6"
format_stamp: "Formatted at 2026-08-10 23:39:29 on dist-test-slave-pww0"
server_key: "2bccc89877cae075a1efa480273a520f"
server_key_iv: "d7a83bf98debe9e8c823b806aea3296a"
server_key_version: "encryptionkey@0"
I20260810 23:39:32.454470 15236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:32.488682 15236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:32.493469 15236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:32.493788 15236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:32.494894 15236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:32.496510 15236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:32.496637 15236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:32.496766 15236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:32.496842 15236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:32.534135 15236 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:45789
I20260810 23:39:32.534215 15471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:45789 every 8 connection(s)
I20260810 23:39:32.535766 15236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:32.539059 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 15236
I20260810 23:39:32.539752 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:36041
--local_ip_for_outbound_sockets=127.30.33.3
--tserver_master_addrs=127.30.33.62:41311
--webserver_port=37807
--webserver_interface=127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:39:32.555521 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:32.538634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:40637 (local address 127.30.33.62:41311)
0810 23:39:32.539715 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:32.539722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:32.540336 (+ 614us) server_negotiation.cc:407] Connection header received
0810 23:39:32.542968 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:32.542973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:32.543051 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:32.543133 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:32.545431 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.546183 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.548533 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.549014 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.549121 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.555273 (+ 6152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:40637: BlockingRecv error: recv got EOF from 127.30.33.2:40637 (error 108)
Metrics: {"server-negotiator.queue_time_us":881}
I20260810 23:39:32.556581 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:32.539182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:40637)
0810 23:39:32.540131 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:32.540192 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:32.542587 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:32.543451 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:32.543491 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:32.544286 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:32.545170 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.545201 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.546399 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.546405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.548345 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.548365 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.549271 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.549279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.549596 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.550403 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:32.550435 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:32.555083 (+ 4648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":368,"threads_started":1}
W20260810 23:39:32.557437 15474 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:41311 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.33.62:41311: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:32.567101 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:32.558083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:39397)
0810 23:39:32.558275 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:32.558295 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:32.558422 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:32.558790 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:32.558794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:32.559223 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:32.559412 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.559425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.560562 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.560567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.561867 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.561886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.562725 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.562731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.562898 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.563567 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:32.563596 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:32.566876 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:39:32.567503 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:32.558266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:39397 (local address 127.30.33.62:41311)
0810 23:39:32.558419 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:32.558423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:32.558442 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:32.558579 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:32.558583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:32.558646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:32.558726 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:32.559631 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.560407 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.562067 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.562546 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.562671 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.567224 (+ 4553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:39397: BlockingRecv error: recv got EOF from 127.30.33.2:39397 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:32.577778 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:32.567965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:37793)
0810 23:39:32.568152 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:32.568173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:32.568304 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:32.568668 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:32.568673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:32.569114 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:32.569350 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.569363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.570541 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.570547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.572079 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:32.572100 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.572873 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.572879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:32.573095 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.573785 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:32.573812 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:32.577417 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 23:39:32.577935 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:32.568132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:37793 (local address 127.30.33.62:41311)
0810 23:39:32.568286 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:32.568291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:32.568309 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:39:32.568436 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:32.568440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:32.568503 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:32.568590 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:32.569502 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.570382 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.572255 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:32.572689 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:32.572817 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:32.577705 (+ 4888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:37793: BlockingRecv error: recv got EOF from 127.30.33.2:37793 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 23:39:32.578289 15474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:39:32.898761 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:32.899077 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:32.899199 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:32.906771 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:32.906895 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:32.907042 15478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:32.917290 15478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/krb5kdc/oryx_127.30.33.3.keytab
--principal=oryx/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:36041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=37807
--tserver_master_addrs=127.30.33.62:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:32.918736 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:32.920164 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:32.933962 15487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:32.934234 15492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:32.933977 15484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:32.935156 15478 server_base.cc:1034] running on GCE node
I20260810 23:39:32.935842 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:32.936805 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:32.938017 15478 hybrid_clock.cc:648] HybridClock initialized: now 1786405172937959 us; error 56 us; skew 500 ppm
I20260810 23:39:32.938341 15478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:32 dist-test-slave-pww0 krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405172, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:32.941713 15478 init.cc:377] Logged in from keytab as oryx/127.30.33.3@KRBTEST.COM (short username oryx)
I20260810 23:39:32.943382 15478 webserver.cc:460] Webserver started at http://127.30.33.3:37807/ using document root <none> and password file <none>
I20260810 23:39:32.944159 15478 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:32.944300 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:32.950979 15478 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260810 23:39:32.954085 15506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:32.955821 15478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:39:32.956009 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "4c30945f1b7841838b5237ca15379a49"
format_stamp: "Formatted at 2026-08-10 23:39:30 on dist-test-slave-pww0"
server_key: "0651371e112e65e28b43a521f35c1fcb"
server_key_iv: "5b087e95c86f6f388627444f06bdd6bf"
server_key_version: "encryptionkey@0"
I20260810 23:39:32.957069 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:32.987922 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:32.993136 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:32.993463 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:32.994633 15478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:32.996206 15478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:32.996313 15478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:32.996428 15478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:32.996500 15478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:33.030787 15478 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:36041
I20260810 23:39:33.030848 15730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:36041 every 8 connection(s)
I20260810 23:39:33.032456 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:33.036568 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 15478
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:39:33.050633 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:33.035307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:40375 (local address 127.30.33.62:41311)
0810 23:39:33.035506 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:33.035513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:33.036967 (+ 1454us) server_negotiation.cc:407] Connection header received
0810 23:39:33.039244 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:33.039250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:33.039354 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:33.039495 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:33.041312 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.042249 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.044163 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.044777 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.044915 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.050334 (+ 5419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:40375: BlockingRecv error: recv got EOF from 127.30.33.3:40375 (error 108)
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":56}
I20260810 23:39:33.051326 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:33.035834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:40375)
0810 23:39:33.036753 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:33.036815 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:33.038949 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:33.039868 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:33.039881 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:33.040455 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:33.041113 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.041134 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.042433 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.042439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.043999 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.044015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.045022 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.045030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.045281 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.045889 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:33.045913 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:33.050054 (+ 4141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":348,"threads_started":1}
W20260810 23:39:33.052103 15732 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:41311 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.33.62:41311: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
I20260810 23:39:33.054725 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:33.037724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.30.33.62:41311)
0810 23:39:33.039927 (+ 2203us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:33.039931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:33.039944 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:39:33.039980 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:33.039983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:33.040028 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:33.040117 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:33.041794 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.042801 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.044215 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.044752 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.044854 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.047891 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:33.047901 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:33.047920 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:33.047959 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:33.051210 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:33.052071 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:33.052078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:33.052085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:33.052302 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:33.052725 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:33.052730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:33.052733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:33.053253 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:33.053577 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:33.054012 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 23:39:33.054404 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":2116,"spinlock_wait_cycles":3968,"thread_start_us":2005,"threads_started":1}
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:33.061084 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:33.052667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:53585)
0810 23:39:33.052864 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:33.052883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:33.052985 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:33.053290 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:33.053294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:33.053631 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:33.053776 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.053785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.054980 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.054985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.056298 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.056312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.057043 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.057049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.057231 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.057838 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:33.057856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:33.060804 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:39:33.061287 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:33.052770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:53585 (local address 127.30.33.62:41311)
0810 23:39:33.052923 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:33.052928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:33.052944 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:39:33.053104 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:33.053109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:33.053169 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:33.053253 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:33.053909 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.054801 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.056435 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.056902 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.057011 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.061057 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:53585: BlockingRecv error: recv got EOF from 127.30.33.3:53585 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:33.070009 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:33.061813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:38129)
0810 23:39:33.062025 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:33.062046 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:33.062155 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:33.062451 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:33.062454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:33.062755 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:33.062913 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.062922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.064010 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.064015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.064983 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.064997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.065764 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.065769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.065909 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.066531 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:33.066550 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:33.069685 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260810 23:39:33.070343 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:33.061942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:38129 (local address 127.30.33.62:41311)
0810 23:39:33.062205 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:33.062210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:33.062229 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:33.062278 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:33.062282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:33.062339 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:33.062419 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:33.063040 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.063859 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.065153 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.065600 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.065711 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.069940 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:38129: BlockingRecv error: recv got EOF from 127.30.33.3:38129 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
W20260810 23:39:33.070477 15732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:33.082566 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:33.073798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:33215)
0810 23:39:33.074305 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:33.074322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:33.074461 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:33.074807 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:33.074811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:33.075300 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:33.075589 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.075601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.076718 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.076722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.077747 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.077761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.078525 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.078557 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.078706 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.079262 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:33.079279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:33.082243 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
W20260810 23:39:33.082731 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:33.073868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:33215 (local address 127.30.33.62:41311)
0810 23:39:33.074081 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:33.074087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:33.074408 (+ 321us) server_negotiation.cc:407] Connection header received
0810 23:39:33.074575 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:33.074579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:33.074646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:33.074727 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:33.075738 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.076564 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.077892 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.078334 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.078453 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.082498 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:33215: BlockingRecv error: recv got EOF from 127.30.33.1:33215 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:33 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:33.588042 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:33.578920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:58689)
0810 23:39:33.579416 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:33.579434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:33.579583 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:33.579953 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:33.579956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:33.580443 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:33.580642 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.580652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.581722 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.581727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.582694 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:33.582707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.583839 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.583844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:33.583972 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.584573 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:33.584623 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:33.587805 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
W20260810 23:39:33.588331 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:33.579013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:58689 (local address 127.30.33.62:41311)
0810 23:39:33.579255 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:33.579262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:33.579537 (+ 275us) server_negotiation.cc:407] Connection header received
0810 23:39:33.579745 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:33.579750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:33.579822 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:33.579917 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:33.580814 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.581589 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.582892 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:33.583627 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:33.583796 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:33.588038 (+ 4242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:58689: BlockingRecv error: recv got EOF from 127.30.33.2:58689 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 23:39:34 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:34 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:34.080430 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:34.071257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:43331)
0810 23:39:34.071869 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:34.071890 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:34.072040 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:34.072422 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:34.072427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:34.072919 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:34.073119 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:34.073129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.074234 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.074238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:34.075227 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:34.075243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.076150 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.076156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:34.076368 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:34.077070 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:34.077090 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:34.080183 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":360,"threads_started":1}
W20260810 23:39:34.080683 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:34.071325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:43331 (local address 127.30.33.62:41311)
0810 23:39:34.071597 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:34.071604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:34.071991 (+ 387us) server_negotiation.cc:407] Connection header received
0810 23:39:34.072172 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:34.072183 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:34.072272 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:34.072361 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:34.073285 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.074073 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.075389 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.075908 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.076052 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:34.080436 (+ 4384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:43331: BlockingRecv error: recv got EOF from 127.30.33.3:43331 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 10 23:39:34 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:34 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:34.092080 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:34.083659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:57329)
0810 23:39:34.084154 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:34.084171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:34.084304 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:34.084654 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:34.084658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:34.085072 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:34.085231 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:34.085242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.086373 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.086377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:34.087296 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:34.087309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.088153 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.088159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:34.088288 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:34.088948 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:34.088969 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:34.091816 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":268,"threads_started":1}
W20260810 23:39:34.092298 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:34.083693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:57329 (local address 127.30.33.62:41311)
0810 23:39:34.083890 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:34.083898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:34.084281 (+ 383us) server_negotiation.cc:407] Connection header received
0810 23:39:34.084459 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:34.084464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:34.084532 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:34.084617 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:34.085402 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.086243 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.087451 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.087962 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.088091 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:34.092031 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:57329: BlockingRecv error: recv got EOF from 127.30.33.1:57329 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 23:39:34 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:34 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:34.600602 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:34.588997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:53543)
0810 23:39:34.589543 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:34.589562 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:34.589686 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:34.590008 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:34.590011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:34.590442 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:34.590601 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:34.590610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.591647 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.591651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:34.592593 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:34.592606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.593408 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.593413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:34.593541 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:34.597416 (+ 3875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:34.597441 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:34.600349 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":300,"threads_started":1}
W20260810 23:39:34.600781 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:34.589070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:53543 (local address 127.30.33.62:41311)
0810 23:39:34.589241 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:34.589245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:34.589631 (+ 386us) server_negotiation.cc:407] Connection header received
0810 23:39:34.589808 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:34.589812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:34.589878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:34.589971 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:34.590732 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.591527 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.592759 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:34.593230 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:34.593340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:34.600566 (+ 7226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:53543: BlockingRecv error: recv got EOF from 127.30.33.2:53543 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:39:35 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:35 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:35.090590 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:35.081419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:48639)
0810 23:39:35.081960 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:35.081982 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:35.082113 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:35.082506 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:35.082512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:35.082995 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:35.083157 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:35.083167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.084367 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.084372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:35.085381 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:35.085396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.086278 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.086283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:35.086406 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:35.087010 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:35.087028 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:35.090228 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
W20260810 23:39:35.090835 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:35.081672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:48639 (local address 127.30.33.62:41311)
0810 23:39:35.081862 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:35.081867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:35.082081 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:39:35.082217 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:35.082221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:35.082309 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:35.082393 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:35.083344 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.084189 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.085550 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.086094 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.086220 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:35.090513 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:48639: BlockingRecv error: recv got EOF from 127.30.33.3:48639 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 23:39:35 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:35 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:35.101986 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:35.093053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:57451)
0810 23:39:35.093624 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:35.093644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:35.093769 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:35.094173 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:35.094177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:35.094533 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:35.094753 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:35.094764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.095920 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.095925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:35.096944 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:35.096961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.097712 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.097718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:35.097871 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:35.098504 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:35.098526 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:35.101712 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":310,"threads_started":1}
W20260810 23:39:35.102411 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:35.093182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:57451 (local address 127.30.33.62:41311)
0810 23:39:35.093388 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:35.093393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:35.093734 (+ 341us) server_negotiation.cc:407] Connection header received
0810 23:39:35.093913 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:35.093918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:35.093987 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:35.094071 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:35.094884 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.095742 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.097103 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.097526 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.097627 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:35.101975 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:57451: BlockingRecv error: recv got EOF from 127.30.33.1:57451 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 23:39:35 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:35 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:35.610594 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:35.601744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:58173)
0810 23:39:35.602346 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:35.602370 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:35.602582 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:35.602975 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:35.602978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:35.603451 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:35.603698 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:35.603709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.604829 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.604834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:35.605776 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:35.605791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.606590 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.606596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:35.606744 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:35.607343 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:35.607363 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:35.610316 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":331,"threads_started":1}
W20260810 23:39:35.610831 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:35.601919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:58173 (local address 127.30.33.62:41311)
0810 23:39:35.602130 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:35.602135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:35.602463 (+ 328us) server_negotiation.cc:407] Connection header received
0810 23:39:35.602741 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:35.602748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:35.602827 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:35.602913 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:35.603829 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.604677 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.605952 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:35.606389 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:35.606496 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:35.610611 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:58173: BlockingRecv error: recv got EOF from 127.30.33.2:58173 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:39:36 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:36 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:36.101027 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:36.091859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:59267)
0810 23:39:36.092449 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:36.092474 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:36.092621 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:36.093000 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:36.093003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:36.093380 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:36.093567 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:36.093576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.094734 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.094738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:36.095775 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:36.095792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.096692 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.096698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:36.096872 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:36.097567 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:36.097588 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:36.100763 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1}
W20260810 23:39:36.101263 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:36.092115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:59267 (local address 127.30.33.62:41311)
0810 23:39:36.092336 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:36.092341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:36.092555 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:39:36.092773 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:36.092778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:36.092874 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:36.092961 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:36.093771 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.094567 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.095954 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.096496 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.096637 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:36.101022 (+ 4385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:59267: BlockingRecv error: recv got EOF from 127.30.33.3:59267 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 10 23:39:36 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:36 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:36.111693 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:36.102954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:37419)
0810 23:39:36.103502 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:36.103523 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:36.103665 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:36.104018 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:36.104023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:36.104417 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:36.104584 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:36.104598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.105634 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.105641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:36.106634 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:36.106648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.107409 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.107415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:36.107639 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:36.108231 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:36.108251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:36.111411 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
W20260810 23:39:36.111881 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:36.103056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:37419 (local address 127.30.33.62:41311)
0810 23:39:36.103283 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:36.103290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:36.103601 (+ 311us) server_negotiation.cc:407] Connection header received
0810 23:39:36.103797 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:36.103802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:36.103869 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:36.103951 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:36.104722 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.105489 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.106787 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.107249 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.107367 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:36.111666 (+ 4299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:37419: BlockingRecv error: recv got EOF from 127.30.33.1:37419 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 23:39:36 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:36 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:36.620134 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:36.611794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:43371)
0810 23:39:36.612344 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:36.612361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:36.612491 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:36.612836 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:36.612839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:36.613251 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:36.613414 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:36.613424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.614432 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.614438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:36.615536 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:36.615554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.616348 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.616354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:36.616543 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:36.617096 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:36.617114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:36.619886 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":277,"threads_started":1}
W20260810 23:39:36.620288 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:36.611965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:43371 (local address 127.30.33.62:41311)
0810 23:39:36.612190 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:36.612196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:36.612444 (+ 248us) server_negotiation.cc:407] Connection header received
0810 23:39:36.612596 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:36.612600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:36.612669 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:36.612785 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:36.613544 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.614270 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.615679 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:36.616193 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:36.616291 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:36.620090 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:43371: BlockingRecv error: recv got EOF from 127.30.33.2:43371 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 23:39:37 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:37 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:37.111099 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:37.102003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:54999)
0810 23:39:37.102546 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:37.102567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:37.102707 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:37.103048 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:37.103053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:37.103491 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:37.103664 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:37.103674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.104957 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.104962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:37.105945 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:37.105959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.106740 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.106746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:37.106922 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:37.107644 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:37.107687 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:37.110772 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":282,"threads_started":1}
W20260810 23:39:37.111287 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:37.102127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:54999 (local address 127.30.33.62:41311)
0810 23:39:37.102357 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:37.102364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:37.102653 (+ 289us) server_negotiation.cc:407] Connection header received
0810 23:39:37.102827 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:37.102832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:37.102897 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:37.102981 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:37.103933 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.104794 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.106107 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.106582 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.106703 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:37.111036 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:54999: BlockingRecv error: recv got EOF from 127.30.33.3:54999 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 23:39:37 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:37 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:37.121758 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:37.112778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:41923)
0810 23:39:37.113440 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:37.113457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:37.113577 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:37.113954 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:37.113958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:37.114462 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:37.114649 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:37.114660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.115800 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.115805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:37.116776 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:37.116789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.117510 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.117515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:37.117663 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:37.118284 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:37.118304 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:37.121493 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":378,"threads_started":1}
W20260810 23:39:37.122030 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:37.112804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:41923 (local address 127.30.33.62:41311)
0810 23:39:37.112994 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:37.113001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:37.113553 (+ 552us) server_negotiation.cc:407] Connection header received
0810 23:39:37.113707 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:37.113712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:37.113781 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:37.113878 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:37.114810 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.115640 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.116897 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.117330 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.117435 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:37.121792 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:41923: BlockingRecv error: recv got EOF from 127.30.33.1:41923 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:39:37 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:37 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:37.630517 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:37.621351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:38817)
0810 23:39:37.621921 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:37.621944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:37.622100 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:37.622467 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:37.622472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:37.622961 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:37.623199 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:37.623214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.624538 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.624544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:37.625492 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:37.625506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.626275 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.626282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:37.626454 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:37.627079 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:37.627100 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:37.630143 (+ 3043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":248,"threads_started":1}
W20260810 23:39:37.630707 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:37.621670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:38817 (local address 127.30.33.62:41311)
0810 23:39:37.621927 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:37.621933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:37.622032 (+ 99us) server_negotiation.cc:407] Connection header received
0810 23:39:37.622241 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:37.622247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:37.622325 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:37.622418 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:37.623371 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.624396 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.625651 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:37.626112 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:37.626229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:37.630438 (+ 4209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:38817: BlockingRecv error: recv got EOF from 127.30.33.2:38817 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 10 23:39:38 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:38 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:38.120391 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:38.112261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:34989)
0810 23:39:38.112937 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:38.112960 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:38.113074 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:38.113341 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:38.113343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:38.113724 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:38.113890 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:38.113898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.114882 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.114885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:38.115884 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:38.115898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.116651 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.116655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:38.116788 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:38.117330 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:38.117348 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:38.120109 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":369,"threads_started":1}
W20260810 23:39:38.120654 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:38.112315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:34989 (local address 127.30.33.62:41311)
0810 23:39:38.112525 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:38.112532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:38.113044 (+ 512us) server_negotiation.cc:407] Connection header received
0810 23:39:38.113163 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:38.113166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:38.113231 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:38.113309 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:38.114009 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.114743 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.116046 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.116511 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.116618 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:38.120371 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:34989: BlockingRecv error: recv got EOF from 127.30.33.3:34989 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 23:39:38 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:38 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:38.131448 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:38.122711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:60787)
0810 23:39:38.123252 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:38.123274 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:38.123427 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:38.123851 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:38.123856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:38.124371 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:38.124576 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:38.124592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.125826 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.125830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:38.126780 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:38.126794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.127507 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.127515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:38.127678 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:38.128290 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:38.128309 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:38.131214 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":263,"threads_started":1}
W20260810 23:39:38.131842 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:38.122828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:60787 (local address 127.30.33.62:41311)
0810 23:39:38.123074 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:38.123080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:38.123363 (+ 283us) server_negotiation.cc:407] Connection header received
0810 23:39:38.123601 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:38.123607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:38.123689 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:38.123785 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:38.124738 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.125675 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.126898 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.127305 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.127403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:38.131568 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:60787: BlockingRecv error: recv got EOF from 127.30.33.1:60787 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 23:39:38 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:38 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:38.640301 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:38.631643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:45481)
0810 23:39:38.632208 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:38.632225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:38.632382 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:38.632777 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:38.632781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:38.633193 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:38.633366 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:38.633376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.634425 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.634429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:38.635331 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:38.635344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.636139 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.636144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:38.636295 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:38.636892 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:38.636911 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:38.639971 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":329,"threads_started":1}
W20260810 23:39:38.640444 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:38.631707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:45481 (local address 127.30.33.62:41311)
0810 23:39:38.631904 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:38.631911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:38.632328 (+ 417us) server_negotiation.cc:407] Connection header received
0810 23:39:38.632512 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:38.632540 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:38.632625 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:38.632723 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:38.633539 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.634279 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.635518 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:38.635977 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:38.636092 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:38.640215 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:45481: BlockingRecv error: recv got EOF from 127.30.33.2:45481 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:39:39 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:39 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:39:39.132580 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:39.121599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:58327 (local address 127.30.33.62:41311)
0810 23:39:39.121817 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:39.121824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:39.122216 (+ 392us) server_negotiation.cc:407] Connection header received
0810 23:39:39.122375 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:39.122381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:39.122451 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:39.122526 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:39.123379 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.124168 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.126098 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.126835 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.126985 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:39.132239 (+ 5254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:58327: BlockingRecv error: recv got EOF from 127.30.33.3:58327 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:39:39.133095 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:39.121475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:58327)
0810 23:39:39.122108 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:39.122130 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:39.122253 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:39.122597 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:39.122602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:39.123037 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:39.123198 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:39.123208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.124349 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.124354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:39.125846 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:39.125868 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.127058 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.127066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:39.127276 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:39.128056 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:39.128081 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:39.132782 (+ 4701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":386,"threads_started":1}
Aug 10 23:39:39 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:39 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:39.142007 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:39.132575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:35279)
0810 23:39:39.133181 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:39.133205 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:39.133351 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:39.133968 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:39.133973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:39.134506 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:39.134746 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:39.134761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.136027 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.136032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:39.137233 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:39.137249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.137886 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.137891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:39.138003 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:39.138521 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:39.138538 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:39.141702 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":263,"threads_started":1}
W20260810 23:39:39.142203 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:39.132683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:35279 (local address 127.30.33.62:41311)
0810 23:39:39.133246 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:39.133251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:39.133278 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:39:39.133524 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:39.133530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:39.133613 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:39.133687 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:39.134948 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.135885 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.137349 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.137765 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.137859 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:39.141956 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:35279: BlockingRecv error: recv got EOF from 127.30.33.1:35279 (error 108)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":696,"threads_started":1}
Aug 10 23:39:39 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:39 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:39.655184 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:39.641474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:35045)
0810 23:39:39.642074 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:39.642092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:39.642229 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:39.642575 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:39.642579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:39.643040 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:39.643216 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:39.643226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.644471 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.644476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:39.645428 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:39.645444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.646199 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.646205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:39.646374 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:39.651130 (+ 4756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:39.651159 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:39.654914 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":274,"threads_started":1}
W20260810 23:39:39.655375 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:39.641508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:35045 (local address 127.30.33.62:41311)
0810 23:39:39.641699 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:39.641705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:39.642183 (+ 478us) server_negotiation.cc:407] Connection header received
0810 23:39:39.642353 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:39.642358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:39.642429 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:39.642511 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:39.643376 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.644313 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.645591 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:39.646033 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:39.646151 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:39.655123 (+ 8972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:35045: BlockingRecv error: recv got EOF from 127.30.33.2:35045 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:39:40 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:40 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:40.142915 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:40.134168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:54389)
0810 23:39:40.134715 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:40.134736 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:40.134874 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:40.135243 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:40.135247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:40.135743 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:40.135925 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:40.135936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.137030 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.137035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:40.137991 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:40.138012 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.138807 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.138813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:40.138981 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:40.139597 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:40.139616 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:40.142587 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":233,"threads_started":1}
W20260810 23:39:40.143373 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:40.134220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:54389 (local address 127.30.33.62:41311)
0810 23:39:40.134397 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:40.134403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:40.134827 (+ 424us) server_negotiation.cc:407] Connection header received
0810 23:39:40.134994 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:40.134999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:40.135087 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:40.135171 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:40.136078 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.136883 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.138156 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.138639 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.138758 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:40.142885 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:54389: BlockingRecv error: recv got EOF from 127.30.33.3:54389 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 23:39:40 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:40 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:40.153040 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:40.143141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:44201)
0810 23:39:40.144214 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:40.144235 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:40.144405 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:40.144806 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:40.144811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:40.145333 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:40.145551 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:40.145565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.146737 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.146741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:40.147817 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:40.147832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.148630 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.148636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:40.148775 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:40.149459 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:40.149482 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:40.152750 (+ 3268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":820,"threads_started":1}
W20260810 23:39:40.153321 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:40.143642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:44201 (local address 127.30.33.62:41311)
0810 23:39:40.143925 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:40.143933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:40.144323 (+ 390us) server_negotiation.cc:407] Connection header received
0810 23:39:40.144544 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:40.144549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:40.144651 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:40.144761 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:40.145696 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.146546 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.147969 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.148411 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.148517 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:40.153046 (+ 4529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:44201: BlockingRecv error: recv got EOF from 127.30.33.1:44201 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Aug 10 23:39:40 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:40 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:40.665699 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:40.656254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:51775)
0810 23:39:40.656876 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:40.656895 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:40.657017 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:40.657439 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:40.657443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:40.657874 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:40.658062 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:40.658072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.659451 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.659486 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:40.660446 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:40.660461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.661287 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.661292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:40.661423 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:40.662181 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:40.662202 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:40.665431 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":359,"threads_started":1}
W20260810 23:39:40.666280 15779 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:40.656440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:51775 (local address 127.30.33.62:41311)
0810 23:39:40.656944 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:40.656949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:40.656968 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:40.657174 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:40.657178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:40.657258 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:40.657366 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:40.658286 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.659300 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.660613 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:40.661105 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:40.661208 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:40.665729 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:51775: BlockingRecv error: recv got EOF from 127.30.33.2:51775 (error 108)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
Aug 10 23:39:41 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:41 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:41.153512 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:41.144172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:45615)
0810 23:39:41.144716 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:41.144738 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:41.144869 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:41.145335 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:41.145339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:41.145899 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:41.146113 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:41.146124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.147420 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.147425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:41.148562 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:41.148579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.149388 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.149395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:41.149555 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:41.150157 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:41.150177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:41.153239 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
W20260810 23:39:41.153710 15779 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:41.144323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:45615 (local address 127.30.33.62:41311)
0810 23:39:41.144610 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:41.144617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:41.144820 (+ 203us) server_negotiation.cc:407] Connection header received
0810 23:39:41.145027 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:41.145032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:41.145142 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:41.145248 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:41.146315 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.147246 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.148732 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.149188 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.149306 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:41.153483 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:45615: BlockingRecv error: recv got EOF from 127.30.33.3:45615 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 10 23:39:41 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:41 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:41.163031 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:41.154074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:56707)
0810 23:39:41.154568 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:41.154584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:41.154720 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:41.155074 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:41.155079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:41.155514 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:41.155719 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:41.155730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.156775 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.156779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:41.157931 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:41.157947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.158723 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.158729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:41.158900 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:41.159566 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:41.159593 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:41.162768 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":259,"threads_started":1}
W20260810 23:39:41.163262 15779 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:41.154127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:56707 (local address 127.30.33.62:41311)
0810 23:39:41.154302 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:41.154309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:41.154671 (+ 362us) server_negotiation.cc:407] Connection header received
0810 23:39:41.154841 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:41.154847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:41.154921 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:41.155007 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:41.155871 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.156628 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.158077 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.158554 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.158681 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:41.163001 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:56707: BlockingRecv error: recv got EOF from 127.30.33.1:56707 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 23:39:41 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:41 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:41.675035 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:41.666678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:48943)
0810 23:39:41.667225 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:41.667248 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:41.667376 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:41.667736 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:41.667739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:41.668129 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:41.668316 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:41.668326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.669456 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.669461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:41.670489 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:41.670506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.671223 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.671227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:41.671361 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:41.671977 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:41.671996 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:41.674774 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":274,"threads_started":1}
W20260810 23:39:41.675263 15784 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:41.666690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:48943 (local address 127.30.33.62:41311)
0810 23:39:41.667350 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:41.667354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:41.667372 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:39:41.667510 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:41.667516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:41.667599 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:41.667688 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:41.668440 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.669294 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.670632 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:41.671090 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:41.671188 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:41.675025 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:48943: BlockingRecv error: recv got EOF from 127.30.33.2:48943 (error 108)
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":452,"threads_started":1}
Aug 10 23:39:42 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:42 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:42.163941 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:42.154632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:36831)
0810 23:39:42.155212 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:42.155234 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:42.155393 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:42.155759 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:42.155764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:42.156186 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:42.156359 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:42.156369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.157485 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.157489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:42.158522 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:42.158546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.159292 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.159298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:42.159513 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:42.160131 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:42.160150 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:42.163641 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":315,"threads_started":1}
W20260810 23:39:42.164122 15784 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:42.154749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:36831 (local address 127.30.33.62:41311)
0810 23:39:42.154960 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:42.154967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:42.155332 (+ 365us) server_negotiation.cc:407] Connection header received
0810 23:39:42.155551 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:42.155555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:42.155624 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:42.155712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:42.156516 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.157335 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.158684 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.159133 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.159249 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:42.163902 (+ 4653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:36831: BlockingRecv error: recv got EOF from 127.30.33.3:36831 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:39:42 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:42 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:42.173034 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:42.164158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:41153)
0810 23:39:42.164678 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:42.164699 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:42.164838 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:42.165168 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:42.165173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:42.165564 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:42.165731 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:42.165741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.166763 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.166769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:42.167804 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:42.167819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.168548 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.168554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:42.168734 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:42.169415 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:42.169436 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:42.172694 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
W20260810 23:39:42.173240 15784 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:42.164495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:41153 (local address 127.30.33.62:41311)
0810 23:39:42.164680 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:42.164686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:42.164775 (+ 89us) server_negotiation.cc:407] Connection header received
0810 23:39:42.164955 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:42.164959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:42.165036 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:42.165118 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:42.165879 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.166620 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.167949 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.168383 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.168496 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:42.172944 (+ 4448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:41153: BlockingRecv error: recv got EOF from 127.30.33.1:41153 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 23:39:42 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:42 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:42.685418 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:42.676176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:50959)
0810 23:39:42.676802 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:42.676822 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:42.676955 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:42.677397 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:42.677402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:42.677835 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:42.677996 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:42.678006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.679160 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.679166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:42.680228 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:42.680249 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.681042 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.681047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:42.681194 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:42.681838 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:42.681856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:42.685083 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":308,"threads_started":1}
W20260810 23:39:42.685649 15790 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:42.676293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:50959 (local address 127.30.33.62:41311)
0810 23:39:42.676814 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:42.676819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:42.676934 (+ 115us) server_negotiation.cc:407] Connection header received
0810 23:39:42.677089 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:42.677095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:42.677190 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:42.677294 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:42.678193 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.678971 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.680411 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:42.680846 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:42.680953 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:42.685382 (+ 4429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:50959: BlockingRecv error: recv got EOF from 127.30.33.2:50959 (error 108)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":274,"threads_started":1}
Aug 10 23:39:43 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:43 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:43.174139 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:43.165339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:60657)
0810 23:39:43.165927 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:43.165952 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:43.166096 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:43.166446 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:43.166451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:43.166872 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:43.167021 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:43.167029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.168180 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.168184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:43.169192 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:43.169211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.169918 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.169924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:43.170099 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:43.170768 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:43.170787 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:43.173784 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
W20260810 23:39:43.174324 15790 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:43.165615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:60657 (local address 127.30.33.62:41311)
0810 23:39:43.165882 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:43.165892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:43.166035 (+ 143us) server_negotiation.cc:407] Connection header received
0810 23:39:43.166229 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:43.166233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:43.166302 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:43.166388 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:43.167181 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.168032 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.169341 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.169760 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.169874 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:43.174076 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:60657: BlockingRecv error: recv got EOF from 127.30.33.3:60657 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Aug 10 23:39:43 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:43 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:43.183842 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:43.174175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:56459)
0810 23:39:43.175276 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:43.175294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:43.175430 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:43.175878 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:43.175882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:43.176372 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:43.176578 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:43.176588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.177726 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.177731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:43.178837 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:43.178856 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.179742 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.179749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:43.179922 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:43.180531 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:43.180551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:43.183446 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":850,"threads_started":1}
W20260810 23:39:43.184026 15790 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:43.174277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:56459 (local address 127.30.33.62:41311)
0810 23:39:43.174622 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:43.174627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:43.175503 (+ 876us) server_negotiation.cc:407] Connection header received
0810 23:39:43.175583 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:43.175587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:43.175666 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:43.175777 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:43.176758 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.177559 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.178978 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.179575 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.179694 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:43.183728 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:56459: BlockingRecv error: recv got EOF from 127.30.33.1:56459 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":502,"threads_started":1}
Aug 10 23:39:43 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:43 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:43.702178 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:43.686741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:45141)
0810 23:39:43.687590 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:43.687618 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:43.687794 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:43.688474 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:43.688478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:43.689150 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:43.689518 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:43.689532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.691172 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.691177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:43.693449 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:43.693480 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.694986 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.695000 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:43.695242 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:43.696242 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:43.696291 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:43.701692 (+ 5401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":406,"threads_started":1}
W20260810 23:39:43.702963 15797 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:43.686938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:45141 (local address 127.30.33.62:41311)
0810 23:39:43.688047 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:43.688052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:43.688093 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:39:43.688163 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:43.688167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:43.688265 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:43.688406 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:43.689744 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.690953 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.693725 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:43.694714 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:43.694894 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:43.702175 (+ 7281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:45141: BlockingRecv error: recv got EOF from 127.30.33.2:45141 (error 108)
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":794,"threads_started":1}
Aug 10 23:39:44 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:44 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:44.183816 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:44.175310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:46603)
0810 23:39:44.175887 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:44.175908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:44.176030 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:44.176388 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:44.176393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:44.176827 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:44.177018 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:44.177026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.178070 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.178074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:44.179018 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:44.179032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.179782 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.179786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:44.179909 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:44.180493 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:44.180511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:44.183428 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":329,"threads_started":1}
W20260810 23:39:44.183966 15797 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:44.175394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:46603 (local address 127.30.33.62:41311)
0810 23:39:44.175598 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:44.175602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:44.175988 (+ 386us) server_negotiation.cc:407] Connection header received
0810 23:39:44.176142 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:44.176146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:44.176226 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:44.176301 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:44.177175 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.177941 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.179162 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.179636 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.179735 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:44.183733 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:46603: BlockingRecv error: recv got EOF from 127.30.33.3:46603 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 23:39:44 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:44 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:44.193449 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:44.184895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:51709)
0810 23:39:44.185377 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:44.185393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:44.185506 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:44.185906 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:44.185909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:44.186399 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:44.186607 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:44.186622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.187849 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.187855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:44.188813 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:44.188829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.189525 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.189528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:44.189659 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:44.190255 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:44.190275 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:44.193184 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
W20260810 23:39:44.193764 15797 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:44.184963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:51709 (local address 127.30.33.62:41311)
0810 23:39:44.185204 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:44.185210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:44.185464 (+ 254us) server_negotiation.cc:407] Connection header received
0810 23:39:44.185637 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:44.185642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:44.185748 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:44.185853 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:44.186781 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.187672 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.188954 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.189408 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.189504 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:44.193495 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:51709: BlockingRecv error: recv got EOF from 127.30.33.1:51709 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:39:44 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:44 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:44.716918 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:44.703517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:43569)
0810 23:39:44.704105 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:44.704124 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:44.704320 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:44.704718 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:44.704723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:44.705131 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:44.705298 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:44.705307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.706421 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.706426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:44.707361 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:44.707376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.708159 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.708165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:44.708372 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:44.713640 (+ 5268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:44.713662 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:44.716672 (+ 3010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":295,"threads_started":1}
W20260810 23:39:44.717247 15808 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:44.703588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:43569 (local address 127.30.33.62:41311)
0810 23:39:44.704104 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:44.704109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:44.704229 (+ 120us) server_negotiation.cc:407] Connection header received
0810 23:39:44.704460 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:44.704465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:44.704547 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:44.704644 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:44.705451 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.706268 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.707542 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:44.707986 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:44.708127 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:44.716966 (+ 8839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:43569: BlockingRecv error: recv got EOF from 127.30.33.2:43569 (error 108)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
Aug 10 23:39:45 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:45 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:45.193622 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:45.184846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:43115)
0810 23:39:45.185443 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:45.185464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:45.185604 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:45.185943 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:45.185948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:45.186417 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:45.186581 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:45.186590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.187641 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.187647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:45.188596 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:45.188608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.189313 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.189320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:45.189502 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:45.190097 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:45.190116 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:45.193286 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":346,"threads_started":1}
W20260810 23:39:45.193769 15808 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:45.184953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:43115 (local address 127.30.33.62:41311)
0810 23:39:45.185219 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:45.185226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:45.185559 (+ 333us) server_negotiation.cc:407] Connection header received
0810 23:39:45.185724 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:45.185728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:45.185793 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:45.185876 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:45.186731 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.187498 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.188742 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.189157 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.189276 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:45.193516 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:43115: BlockingRecv error: recv got EOF from 127.30.33.3:43115 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 10 23:39:45 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:45 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:45.202453 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:45.194481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:55597)
0810 23:39:45.194991 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:45.195008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:45.195122 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:45.195412 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:45.195415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:45.195825 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:45.195997 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:45.196006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.196976 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.196979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:45.197904 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:45.197916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.198621 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.198627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:45.198756 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:45.199292 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:45.199310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:45.202099 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
W20260810 23:39:45.202598 15808 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:45.194519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:55597 (local address 127.30.33.62:41311)
0810 23:39:45.194709 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:45.194716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:45.195081 (+ 365us) server_negotiation.cc:407] Connection header received
0810 23:39:45.195228 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:45.195233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:45.195301 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:45.195372 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:45.196122 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.196866 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.198020 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.198434 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.198524 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:45.202356 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:55597: BlockingRecv error: recv got EOF from 127.30.33.1:55597 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 23:39:45 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:45 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:45.726464 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:45.717954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:56373)
0810 23:39:45.718507 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:45.718525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:45.718657 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:45.719025 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:45.719028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:45.719443 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:45.719661 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:45.719672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.720680 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.720684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:45.721641 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:45.721654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.722347 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.722351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:45.722468 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:45.723079 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:45.723099 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:45.726212 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":276,"threads_started":1}
W20260810 23:39:45.726686 15817 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:45.718146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:56373 (local address 127.30.33.62:41311)
0810 23:39:45.718758 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:45.718763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:45.718779 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:39:45.718828 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:45.718832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:45.718905 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:45.718993 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:45.719797 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.720547 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.721764 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:45.722198 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:45.722289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:45.726457 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:56373: BlockingRecv error: recv got EOF from 127.30.33.2:56373 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":386,"threads_started":1}
Aug 10 23:39:46 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:46 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:46.203748 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:46.194792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:39153)
0810 23:39:46.195393 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:46.195415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:46.195624 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:46.195987 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:46.195991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:46.196426 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:46.196642 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:46.196653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.197724 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.197730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:46.198745 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:46.198761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.199572 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.199579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:46.199712 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:46.200303 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:46.200323 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:46.203367 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":309,"threads_started":1}
W20260810 23:39:46.203922 15817 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:46.194932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:39153 (local address 127.30.33.62:41311)
0810 23:39:46.195119 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:46.195123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:46.195518 (+ 395us) server_negotiation.cc:407] Connection header received
0810 23:39:46.195739 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:46.195745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:46.195815 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:46.195910 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:46.196775 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.197541 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.198879 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.199319 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.199417 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:46.203644 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:39153: BlockingRecv error: recv got EOF from 127.30.33.3:39153 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:39:46 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:46 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:46.212509 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:46.203494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:44133)
0810 23:39:46.204081 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:46.204103 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:46.204243 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:46.204699 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:46.204704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:46.205211 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:46.205438 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:46.205448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.206586 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.206590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:46.207600 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:46.207615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.208341 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.208346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:46.208483 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:46.209099 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:46.209119 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:46.212230 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":820,"threads_started":1}
W20260810 23:39:46.212749 15817 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:46.203879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:44133 (local address 127.30.33.62:41311)
0810 23:39:46.204153 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:46.204158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:46.204177 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:46.204415 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:46.204422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:46.204511 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:46.204628 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:46.205623 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.206432 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.207743 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.208171 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.208278 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:46.212464 (+ 4186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:44133: BlockingRecv error: recv got EOF from 127.30.33.1:44133 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":524,"threads_started":1}
Aug 10 23:39:46 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:46 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:46.736544 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:46.727639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:48715)
0810 23:39:46.728144 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:46.728162 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:46.728287 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:46.728610 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:46.728613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:46.729022 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:46.729214 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:46.729224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.730662 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.730669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:46.731739 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:46.731756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.732465 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.732470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:46.732598 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:46.733249 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:46.733268 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:46.736216 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
W20260810 23:39:46.736726 15825 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:46.727768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:48715 (local address 127.30.33.62:41311)
0810 23:39:46.728254 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:46.728259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:46.728281 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:39:46.728401 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:46.728406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:46.728470 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:46.728578 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:46.729396 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.730473 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.731877 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:46.732334 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:46.732431 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:46.736486 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:48715: BlockingRecv error: recv got EOF from 127.30.33.2:48715 (error 108)
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":263,"threads_started":1}
Aug 10 23:39:47 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:47 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.3@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:47.214112 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:47.204948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.3:60967)
0810 23:39:47.205538 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:47.205560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:47.205716 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:47.206090 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:47.206095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:47.206529 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:47.206732 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:47.206743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.207948 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.207952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:47.208893 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:47.208907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.209657 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.209663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:47.209841 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:47.210480 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:47.210499 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:47.213759 (+ 3260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":365,"threads_started":1}
W20260810 23:39:47.214354 15825 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:47.205106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:60967 (local address 127.30.33.62:41311)
0810 23:39:47.205312 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:47.205319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:47.205666 (+ 347us) server_negotiation.cc:407] Connection header received
0810 23:39:47.205859 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:47.205864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:47.205937 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:47.206026 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:47.206910 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.207782 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.209056 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.209496 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.209616 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:47.214045 (+ 4429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.3:60967: BlockingRecv error: recv got EOF from 127.30.33.3:60967 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 23:39:47 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:47 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.1@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:47.224213 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:47.213865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.1:36739)
0810 23:39:47.215433 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:47.215482 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:47.215632 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:47.215995 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:47.215998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:47.216428 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:47.216643 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:47.216654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.217854 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.217859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:47.218855 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:47.218872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.219707 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.219713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:47.219861 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:47.220510 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:47.220531 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:47.223918 (+ 3387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1447,"thread_start_us":1335,"threads_started":1}
W20260810 23:39:47.224404 15825 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:47.213958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:36739 (local address 127.30.33.62:41311)
0810 23:39:47.214961 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:47.214968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:47.215572 (+ 604us) server_negotiation.cc:407] Connection header received
0810 23:39:47.215744 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:47.215748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:47.215815 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:47.215901 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:47.216838 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.217674 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.219024 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.219540 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.219663 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:47.224153 (+ 4490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.1:36739: BlockingRecv error: recv got EOF from 127.30.33.1:36739 (error 108)
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":1004,"threads_started":1}
Aug 10 23:39:47 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:39:47 dist-test-slave-pww0 krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.33.2@KRBTEST.COM for oryx/127.30.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:39:47.746798 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:47.737918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41311 (local address 127.30.33.2:60737)
0810 23:39:47.738518 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:47.738541 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:47.738675 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:47.739191 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:47.739196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:47.739724 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:47.739935 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:47.739946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.741074 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.741079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:47.742152 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:47.742167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.742886 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.742892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:47.743063 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:47.743690 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:47.743709 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:47.746520 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.33.62:41311: Server oryx/127.30.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":310,"threads_started":1}
W20260810 23:39:47.747053 15835 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:47.738087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:60737 (local address 127.30.33.62:41311)
0810 23:39:47.738830 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:47.738834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:47.738853 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:39:47.738909 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:47.738912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:47.738973 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:47.739068 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:47.740108 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.740905 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.742296 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:47.742738 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:47.742855 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:47.746782 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.33.2:60737: BlockingRecv error: recv got EOF from 127.30.33.2:60737 (error 108)
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":475,"threads_started":1}
I20260810 23:39:48.043020 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 14950
I20260810 23:39:48.056221 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:48.044366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.30.33.1:43383)
0810 23:39:48.044964 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:48.044973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:48.045006 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:39:48.045074 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:48.045081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:48.045324 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:48.045492 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:48.047111 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.048223 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.049518 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.050120 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.050236 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:48.053047 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:48.053057 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:48.053073 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:48.053118 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:48.055636 (+ 2518us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:39:48.055977 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58014: Request ticket server kudu/127.30.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":302,"threads_started":1}
W20260810 23:39:48.056340 15838 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58014: Request ticket server kudu/127.30.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:39:48.056483 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 14950: Not authorized: Client connection negotiation failed: client connection to 127.30.33.1:43383: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.33.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:39:48.056742 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 14950
I20260810 23:39:48.091632 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 15236
I20260810 23:39:48.105793 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:48.092814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.30.33.2:45789)
0810 23:39:48.093485 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:48.093493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:48.093525 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:39:48.093592 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:48.093599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:48.093858 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:48.094021 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:48.095704 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.096852 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.098225 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.098817 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.098929 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:48.102178 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:48.102191 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:48.102208 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:48.102274 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:48.105171 (+ 2897us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:39:48.105534 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60786: Request ticket server kudu/127.30.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":328,"threads_started":1}
W20260810 23:39:48.105918 15839 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60786: Request ticket server kudu/127.30.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:39:48.106122 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 15236: Not authorized: Client connection negotiation failed: client connection to 127.30.33.2:45789: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.33.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:39:48.106297 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 15236
I20260810 23:39:48.141395 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 15478
I20260810 23:39:48.154003 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:48.142338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.30.33.3:36041)
0810 23:39:48.142908 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:48.142916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:48.142946 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:39:48.143013 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:48.143019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:48.143278 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:48.143489 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:48.144807 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.145867 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.147200 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.147873 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.147988 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:48.150622 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:48.150631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:48.150647 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:48.150696 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:48.153435 (+ 2739us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:39:48.153751 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44506: Request ticket server kudu/127.30.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":280,"threads_started":1}
W20260810 23:39:48.154106 15841 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44506: Request ticket server kudu/127.30.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:39:48.154367 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 15478: Not authorized: Client connection negotiation failed: client connection to 127.30.33.3:36041: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.33.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:39:48.154533 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 15478
I20260810 23:39:48.179450 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 14821
I20260810 23:39:48.180929 14935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:48.303488 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 14821
2026-08-10T23:39:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21843 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T23:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:48Z Disabled control of system clock
I20260810 23:39:48.407136 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35085
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:35085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:39:48.777766 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:48.778106 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:48.778192 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:48.785697 15853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:48.785830 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:48.785871 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:48.785928 15853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:48.785974 15853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:48.795763 15853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:35085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:48.797187 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:48.799160 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:48.811990 15864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:48.812031 15865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:48.812031 15867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:48.812707 15853 server_base.cc:1034] running on GCE node
I20260810 23:39:48.813683 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:48.815326 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:48.816596 15853 hybrid_clock.cc:648] HybridClock initialized: now 1786405188816524 us; error 91 us; skew 500 ppm
I20260810 23:39:48.817036 15853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:48.819832 15853 webserver.cc:460] Webserver started at http://127.30.33.62:34005/ using document root <none> and password file <none>
I20260810 23:39:48.820650 15853 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:48.820780 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:48.821159 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:48.824527 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "deb4b6c59e264eba9a73441c8ce5cdf2"
format_stamp: "Formatted at 2026-08-10 23:39:48 on dist-test-slave-pww0"
server_key: "2ed9ab45a73ca40c8b95ca4f0432893e"
server_key_iv: "900ef9505ff6d409a10a26973c806e3e"
server_key_version: "encryptionkey@0"
I20260810 23:39:48.825449 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "deb4b6c59e264eba9a73441c8ce5cdf2"
format_stamp: "Formatted at 2026-08-10 23:39:48 on dist-test-slave-pww0"
server_key: "2ed9ab45a73ca40c8b95ca4f0432893e"
server_key_iv: "900ef9505ff6d409a10a26973c806e3e"
server_key_version: "encryptionkey@0"
I20260810 23:39:48.831794 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 23:39:48.835870 15880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:48.837425 15853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:39:48.837635 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "deb4b6c59e264eba9a73441c8ce5cdf2"
format_stamp: "Formatted at 2026-08-10 23:39:48 on dist-test-slave-pww0"
server_key: "2ed9ab45a73ca40c8b95ca4f0432893e"
server_key_iv: "900ef9505ff6d409a10a26973c806e3e"
server_key_version: "encryptionkey@0"
I20260810 23:39:48.837873 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:48.857651 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:48.858745 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:48.859076 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:48.878377 15853 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:35085
I20260810 23:39:48.878397 15975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:35085 every 8 connection(s)
I20260810 23:39:48.879904 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:48.883976 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:48.888837 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 15853
I20260810 23:39:48.889025 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:48.889711 30852 external_mini_cluster.cc:1428] Setting key 04f3816f8d168e26a1bfe0652e18a314
I20260810 23:39:48.897157 15979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2: Bootstrap starting.
I20260810 23:39:48.901221 15979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:48.902720 15979 log.cc:824] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:48.906739 15979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2: No bootstrap required, opened a new log
I20260810 23:39:48.907361 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:48.892162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.30.33.62:35085)
0810 23:39:48.893429 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:48.893447 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:48.893512 (+ 65us) server_negotiation.cc:407] Connection header received
0810 23:39:48.896038 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:48.896080 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:48.896810 (+ 730us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:48.897333 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:48.897946 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.899986 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.901550 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:48.902733 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:48.902930 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:48.904251 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:48.904260 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:48.904277 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:48.905045 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:48.905157 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:48.905188 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:48.905315 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:48.905583 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:39:48.905958 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":374,"threads_started":1}
I20260810 23:39:48.915541 15979 raft_consensus.cc:348] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35085 } }
I20260810 23:39:48.915962 15979 raft_consensus.cc:374] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:48.916042 15979 raft_consensus.cc:729] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb4b6c59e264eba9a73441c8ce5cdf2, State: Initialized, Role: FOLLOWER
I20260810 23:39:48.916630 15979 consensus_queue.cc:260] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35085 } }
I20260810 23:39:48.916859 15979 raft_consensus.cc:388] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:48.916967 15979 raft_consensus.cc:482] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:48.917109 15979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:48.919359 15979 raft_consensus.cc:504] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35085 } }
I20260810 23:39:48.919955 15979 leader_election.cc:302] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deb4b6c59e264eba9a73441c8ce5cdf2; no voters:
I20260810 23:39:48.920543 15979 leader_election.cc:288] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:48.920761 15992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:48.921412 15992 raft_consensus.cc:686] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [term 1 LEADER]: Becoming Leader. State: Replica: deb4b6c59e264eba9a73441c8ce5cdf2, State: Running, Role: LEADER
I20260810 23:39:48.922127 15992 consensus_queue.cc:237] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35085 } }
I20260810 23:39:48.923856 15979 sys_catalog.cc:564] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:48.925185 15997 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35085 } } }
I20260810 23:39:48.925639 15997 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:48.926156 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deb4b6c59e264eba9a73441c8ce5cdf2. Latest consensus state: current_term: 1 leader_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb4b6c59e264eba9a73441c8ce5cdf2" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35085 } } }
I20260810 23:39:48.926493 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:48.926725 16017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:48.933699 16017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:48.943286 16017 catalog_manager.cc:1347] Generated new cluster ID: 5b28669cd17047f7b72d42de697b2e28
I20260810 23:39:48.943434 16017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:48.960634 16017 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:48.962069 16017 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:48.968722 16017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deb4b6c59e264eba9a73441c8ce5cdf2: Generated new TSK 0
I20260810 23:39:48.969933 16017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:39:48.982319 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35085
--builtin_ntp_servers=127.30.33.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 23:39:49.378223 16037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:49.378542 16037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:49.378654 16037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:49.386234 16037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:49.386363 16037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:49.386509 16037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:49.396092 16037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:49.397480 16037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:49.399389 16037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:49.413615 16047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:49.413595 16048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:49.413561 16051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:49.414528 16037 server_base.cc:1034] running on GCE node
I20260810 23:39:49.415244 16037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:49.416379 16037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:49.417595 16037 hybrid_clock.cc:648] HybridClock initialized: now 1786405189417541 us; error 53 us; skew 500 ppm
I20260810 23:39:49.417951 16037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:49.420461 16037 webserver.cc:460] Webserver started at http://127.30.33.1:32863/ using document root <none> and password file <none>
I20260810 23:39:49.421243 16037 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:49.421360 16037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:49.421717 16037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:49.425105 16037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "0baa42a665764d9797b825f5bf51f3b4"
format_stamp: "Formatted at 2026-08-10 23:39:49 on dist-test-slave-pww0"
server_key: "991084e4679b1d54733ebe2a603f7879"
server_key_iv: "a9ef18ef90386d13802241db19dfc95e"
server_key_version: "encryptionkey@0"
I20260810 23:39:49.425985 16037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "0baa42a665764d9797b825f5bf51f3b4"
format_stamp: "Formatted at 2026-08-10 23:39:49 on dist-test-slave-pww0"
server_key: "991084e4679b1d54733ebe2a603f7879"
server_key_iv: "a9ef18ef90386d13802241db19dfc95e"
server_key_version: "encryptionkey@0"
I20260810 23:39:49.432190 16037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:39:49.436291 16060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:49.437976 16037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:39:49.438184 16037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "0baa42a665764d9797b825f5bf51f3b4"
format_stamp: "Formatted at 2026-08-10 23:39:49 on dist-test-slave-pww0"
server_key: "991084e4679b1d54733ebe2a603f7879"
server_key_iv: "a9ef18ef90386d13802241db19dfc95e"
server_key_version: "encryptionkey@0"
I20260810 23:39:49.438427 16037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:49.451277 16037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:49.452283 16037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:49.452587 16037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:49.453692 16037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:49.455353 16037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:49.455510 16037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:49.455624 16037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:49.455703 16037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:49.488152 16037 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:35695
I20260810 23:39:49.488229 16267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:35695 every 8 connection(s)
I20260810 23:39:49.489602 16037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:49.490149 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 16037
I20260810 23:39:49.490357 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:39:49.491163 30852 external_mini_cluster.cc:1428] Setting key b33aaece4db1377e591494004a155253
I20260810 23:39:49.494395 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35085
--builtin_ntp_servers=127.30.33.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:39:49.504158 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:49.492494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:34531 (local address 127.30.33.62:35085)
0810 23:39:49.492989 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:49.492994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:49.495544 (+ 2550us) server_negotiation.cc:407] Connection header received
0810 23:39:49.496725 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:49.496732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:49.496847 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:49.496984 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:49.498320 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:49.499277 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:49.501103 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:49.501660 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:49.501811 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:49.503244 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:49.503253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:49.503257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:49.503284 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:49.503336 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:49.503353 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:49.503444 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:49.503693 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 23:39:49.503802 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260810 23:39:49.505192 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:49.493338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35085 (local address 127.30.33.1:34531)
0810 23:39:49.494569 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:49.494619 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:49.496397 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:49.497272 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:49.497283 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:49.497413 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:49.498099 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:49.498121 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:49.499540 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:49.499545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:49.500925 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:49.500941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:49.501879 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:49.501884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:49.502116 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:49.502860 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:39:49.502863 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:49.502915 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:39:49.503007 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:49.503444 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:49.503451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:49.503483 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:39:49.503655 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 23:39:49.504023 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":369,"threads_started":1}
I20260810 23:39:49.506947 16270 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35085
I20260810 23:39:49.507313 16270 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:49.508196 16270 heartbeater.cc:511] Master 127.30.33.62:35085 requested a full tablet report, sending...
I20260810 23:39:49.510803 15917 ts_manager.cc:194] Registered new tserver with Master: 0baa42a665764d9797b825f5bf51f3b4 (127.30.33.1:35695)
I20260810 23:39:49.512990 15917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.1:34531
W20260810 23:39:49.865487 16278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:49.865803 16278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:49.865891 16278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:49.873322 16278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:49.873448 16278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:49.873555 16278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:49.886620 16278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:49.888505 16278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:49.890950 16278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:49.906224 16288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:49.906164 16295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:49.906242 16289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:49.907349 16278 server_base.cc:1034] running on GCE node
I20260810 23:39:49.908113 16278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:49.909255 16278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:49.910507 16278 hybrid_clock.cc:648] HybridClock initialized: now 1786405189910453 us; error 46 us; skew 500 ppm
I20260810 23:39:49.910917 16278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:49.918094 16278 webserver.cc:460] Webserver started at http://127.30.33.2:39603/ using document root <none> and password file <none>
I20260810 23:39:49.918956 16278 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:49.919082 16278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:49.919440 16278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:49.922676 16278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "e0d20a02fe4547999e7f0dc12eebad63"
format_stamp: "Formatted at 2026-08-10 23:39:49 on dist-test-slave-pww0"
server_key: "e47612d5c9ca6337002dcde1252ed391"
server_key_iv: "d48fff5bbe8e7d462e28f3eb8f9e6a0a"
server_key_version: "encryptionkey@0"
I20260810 23:39:49.923671 16278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "e0d20a02fe4547999e7f0dc12eebad63"
format_stamp: "Formatted at 2026-08-10 23:39:49 on dist-test-slave-pww0"
server_key: "e47612d5c9ca6337002dcde1252ed391"
server_key_iv: "d48fff5bbe8e7d462e28f3eb8f9e6a0a"
server_key_version: "encryptionkey@0"
I20260810 23:39:49.929804 16278 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:39:49.933903 16306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:49.935580 16278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:39:49.935765 16278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "e0d20a02fe4547999e7f0dc12eebad63"
format_stamp: "Formatted at 2026-08-10 23:39:49 on dist-test-slave-pww0"
server_key: "e47612d5c9ca6337002dcde1252ed391"
server_key_iv: "d48fff5bbe8e7d462e28f3eb8f9e6a0a"
server_key_version: "encryptionkey@0"
I20260810 23:39:49.935998 16278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:49.951349 16278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:49.952360 16278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:49.952647 16278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:49.953759 16278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:49.955423 16278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:49.955706 16278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:39:49.955885 16278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:49.955979 16278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:49.988330 16278 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:42757
I20260810 23:39:49.988405 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:42757 every 8 connection(s)
I20260810 23:39:49.989763 16278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:50.000032 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 16278
I20260810 23:39:50.000234 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:39:50.000797 30852 external_mini_cluster.cc:1428] Setting key ce5c38ffe3e0491d2a07e7cb0f04f9bb
I20260810 23:39:50.003212 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:49.992590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:53385 (local address 127.30.33.62:35085)
0810 23:39:49.992825 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:49.992831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:49.994436 (+ 1605us) server_negotiation.cc:407] Connection header received
0810 23:39:49.996377 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:49.996382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:49.996462 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:49.996561 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:49.997696 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:49.998433 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.000152 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.000784 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.000940 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:50.002362 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:50.002369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:50.002373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:50.002395 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:50.002437 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:50.002453 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:50.002553 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:50.002838 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 23:39:50.002946 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:39:50.004125 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35085
--builtin_ntp_servers=127.30.33.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:39:50.004793 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:49.993152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35085 (local address 127.30.33.2:53385)
0810 23:39:49.994253 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:49.994302 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:49.996070 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:49.996804 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:49.996814 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:49.996913 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:49.997551 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:49.997569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:49.998565 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:49.998569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:49.999974 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:49.999987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.001027 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.001032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:50.001261 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:50.001993 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:39:50.001996 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:50.002044 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:39:50.002136 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:50.002549 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:50.002559 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:50.002567 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:39:50.002798 (+ 231us) client_negotiation.cc:240] Negotiation successful
0810 23:39:50.003251 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":359,"threads_started":1}
I20260810 23:39:50.006455 16531 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35085
I20260810 23:39:50.006784 16531 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:50.007700 16531 heartbeater.cc:511] Master 127.30.33.62:35085 requested a full tablet report, sending...
I20260810 23:39:50.009303 15917 ts_manager.cc:194] Registered new tserver with Master: e0d20a02fe4547999e7f0dc12eebad63 (127.30.33.2:42757)
I20260810 23:39:50.010164 15917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.2:53385
W20260810 23:39:50.357868 16535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:50.358179 16535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:50.358304 16535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:50.365952 16535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:50.366086 16535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:50.366228 16535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:50.375901 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:35085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:50.377341 16535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:50.379247 16535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:50.393556 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:50.393710 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:50.393566 16542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:50.394397 16535 server_base.cc:1034] running on GCE node
I20260810 23:39:50.395146 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:50.396245 16535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:50.397455 16535 hybrid_clock.cc:648] HybridClock initialized: now 1786405190397401 us; error 44 us; skew 500 ppm
I20260810 23:39:50.397782 16535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:50.400471 16535 webserver.cc:460] Webserver started at http://127.30.33.3:45291/ using document root <none> and password file <none>
I20260810 23:39:50.401258 16535 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:50.401373 16535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:50.401757 16535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:50.404953 16535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "b736b6cea98d485cbc444ea3b3bf6f87"
format_stamp: "Formatted at 2026-08-10 23:39:50 on dist-test-slave-pww0"
server_key: "45e14dcac3ba842acf0a2e49f8f555f9"
server_key_iv: "5eb0cf3762f5a78f6dc4ce3caddc531a"
server_key_version: "encryptionkey@0"
I20260810 23:39:50.405901 16535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "b736b6cea98d485cbc444ea3b3bf6f87"
format_stamp: "Formatted at 2026-08-10 23:39:50 on dist-test-slave-pww0"
server_key: "45e14dcac3ba842acf0a2e49f8f555f9"
server_key_iv: "5eb0cf3762f5a78f6dc4ce3caddc531a"
server_key_version: "encryptionkey@0"
I20260810 23:39:50.411715 16535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:39:50.415632 16564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:50.417181 16535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:39:50.417382 16535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "b736b6cea98d485cbc444ea3b3bf6f87"
format_stamp: "Formatted at 2026-08-10 23:39:50 on dist-test-slave-pww0"
server_key: "45e14dcac3ba842acf0a2e49f8f555f9"
server_key_iv: "5eb0cf3762f5a78f6dc4ce3caddc531a"
server_key_version: "encryptionkey@0"
I20260810 23:39:50.417623 16535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:50.437697 16535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:50.438675 16535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:50.438970 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:50.440105 16535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:50.441732 16535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:50.441854 16535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:50.441963 16535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:50.442046 16535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:50.474273 16535 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:37291
I20260810 23:39:50.474323 16790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:37291 every 8 connection(s)
I20260810 23:39:50.475775 16535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:50.476486 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 16535
I20260810 23:39:50.476701 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:39:50.477262 30852 external_mini_cluster.cc:1428] Setting key 6fcb67e0e990ae00e5200463d2df7fd3
I20260810 23:39:50.489740 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:50.478785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:37215 (local address 127.30.33.62:35085)
0810 23:39:50.478960 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:50.478965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:50.480739 (+ 1774us) server_negotiation.cc:407] Connection header received
0810 23:39:50.482679 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:50.482685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:50.482783 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:50.482876 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:50.484187 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.484954 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.486627 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.487045 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.487179 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:50.488834 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:50.488840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:50.488843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:50.488865 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:50.488901 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:50.488913 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:50.488996 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:50.489257 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:39:50.489379 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:39:50.490854 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:50.479409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:35085 (local address 127.30.33.3:37215)
0810 23:39:50.480560 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:50.480626 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:50.482372 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:50.483154 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:50.483165 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:50.483273 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:50.483993 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:50.484014 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.485116 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.485120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:50.486462 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:50.486476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.487294 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.487302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:50.487616 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:50.488452 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:39:50.488455 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:50.488501 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:39:50.488598 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:50.489008 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:50.489022 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:50.489031 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:39:50.489195 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 23:39:50.489668 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":356,"threads_started":1}
I20260810 23:39:50.492836 16791 heartbeater.cc:347] Connected to a master server at 127.30.33.62:35085
I20260810 23:39:50.493149 16791 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:50.494117 16791 heartbeater.cc:511] Master 127.30.33.62:35085 requested a full tablet report, sending...
I20260810 23:39:50.495738 15912 ts_manager.cc:194] Registered new tserver with Master: b736b6cea98d485cbc444ea3b3bf6f87 (127.30.33.3:37291)
I20260810 23:39:50.496649 15912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.3:37215
I20260810 23:39:50.506880 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:39:50.513680 16275 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:50.511859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.30.33.62:35085)
0810 23:39:50.512090 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:50.512097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:50.512547 (+ 450us) server_negotiation.cc:407] Connection header received
0810 23:39:50.512739 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:50.512744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:50.512813 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:50.512907 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:50.513407 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52642: BlockingRecv error: recv got EOF from 127.0.0.1:52642 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:39:50.514489 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 16037
I20260810 23:39:50.516530 16270 heartbeater.cc:503] Master 127.30.33.62:35085 was elected leader, sending a full tablet report...
I20260810 23:39:50.523232 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:50.515521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.30.33.1:35695)
0810 23:39:50.516127 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:50.516134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:50.516163 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:39:50.516226 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:50.516232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:50.516505 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:50.516662 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:50.517130 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.518291 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.519702 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.520297 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.520419 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:50.521640 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:50.521650 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:50.521666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:50.522329 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:50.522400 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:50.522425 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:50.522535 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:50.522744 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 23:39:50.522875 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"spinlock_wait_cycles":42368,"thread_start_us":289,"threads_started":1}
I20260810 23:39:50.524686 16262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:50.608876 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 16037
I20260810 23:39:50.632207 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 16278
I20260810 23:39:50.640873 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:50.633297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.30.33.2:42757)
0810 23:39:50.633933 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:50.633940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:50.633971 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:39:50.634058 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:50.634064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:50.634312 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:50.634447 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:50.634842 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.635955 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.637282 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.637922 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.638042 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:50.639101 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:50.639111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:50.639128 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:50.639960 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:50.640021 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:50.640047 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:50.640160 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:50.640381 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 23:39:50.640525 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":340,"threads_started":1}
I20260810 23:39:50.642453 16525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:50.727749 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 16278
I20260810 23:39:50.754698 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 16535
I20260810 23:39:50.763574 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:50.755811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.30.33.3:37291)
0810 23:39:50.756385 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:50.756393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:50.756426 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:39:50.756494 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:50.756501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:50.756731 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:50.756869 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:50.757438 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.758551 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.760099 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:50.760734 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:50.760847 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:50.762083 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:50.762093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:50.762108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:50.762754 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:50.762817 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:50.762840 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:50.762937 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:50.763047 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:39:50.763208 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":269,"threads_started":1}
I20260810 23:39:50.764940 16780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:50.846576 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 16535
I20260810 23:39:50.867781 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 15853
I20260810 23:39:50.869011 15968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:50.974987 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 15853
2026-08-10T23:39:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2670 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T23:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:51Z Disabled control of system clock
I20260810 23:39:51.078934 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46507
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:46507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:39:51.439486 16831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:51.439842 16831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:51.439935 16831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:51.447539 16831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:51.447677 16831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:51.447722 16831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:51.447781 16831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:51.447839 16831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:51.457953 16831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:46507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:51.459393 16831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:51.461380 16831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:51.473786 16839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:51.473866 16838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:51.473806 16842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:51.474805 16831 server_base.cc:1034] running on GCE node
I20260810 23:39:51.475729 16831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:51.477252 16831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:51.478495 16831 hybrid_clock.cc:648] HybridClock initialized: now 1786405191478431 us; error 46 us; skew 500 ppm
I20260810 23:39:51.478861 16831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:51.481350 16831 webserver.cc:460] Webserver started at http://127.30.33.62:45479/ using document root <none> and password file <none>
I20260810 23:39:51.482086 16831 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:51.482182 16831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:51.482474 16831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:51.485594 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "175e6488a8d644e6b44f3ecc775877b8"
format_stamp: "Formatted at 2026-08-10 23:39:51 on dist-test-slave-pww0"
server_key: "91894221f368edd590c3a8371ba213a1"
server_key_iv: "829b6ae6daf1bbf77ff04caa36d34926"
server_key_version: "encryptionkey@0"
I20260810 23:39:51.486403 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "175e6488a8d644e6b44f3ecc775877b8"
format_stamp: "Formatted at 2026-08-10 23:39:51 on dist-test-slave-pww0"
server_key: "91894221f368edd590c3a8371ba213a1"
server_key_iv: "829b6ae6daf1bbf77ff04caa36d34926"
server_key_version: "encryptionkey@0"
I20260810 23:39:51.492077 16831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:39:51.496029 16861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:51.497606 16831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:39:51.497781 16831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "175e6488a8d644e6b44f3ecc775877b8"
format_stamp: "Formatted at 2026-08-10 23:39:51 on dist-test-slave-pww0"
server_key: "91894221f368edd590c3a8371ba213a1"
server_key_iv: "829b6ae6daf1bbf77ff04caa36d34926"
server_key_version: "encryptionkey@0"
I20260810 23:39:51.497994 16831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:51.504242 16831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:51.505125 16831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:51.505420 16831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:51.524251 16831 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:46507
I20260810 23:39:51.524287 16951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:46507 every 8 connection(s)
I20260810 23:39:51.525710 16831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:51.529902 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:51.530920 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 16831
I20260810 23:39:51.531132 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:51.531847 30852 external_mini_cluster.cc:1428] Setting key bba3680bd942c7ffbae9821d3188398b
I20260810 23:39:51.541797 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:51.534120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.30.33.62:46507)
0810 23:39:51.535359 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:51.535376 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:51.535440 (+ 64us) server_negotiation.cc:407] Connection header received
0810 23:39:51.537190 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:51.537234 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:51.537918 (+ 684us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:51.538445 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:51.539590 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:51.539599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:51.539616 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:51.539637 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:51.539726 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:51.539761 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:51.540002 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:51.540132 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 23:39:51.540431 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":370,"threads_started":1}
I20260810 23:39:51.545454 16953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8: Bootstrap starting.
I20260810 23:39:51.548715 16953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:51.549839 16953 log.cc:824] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:51.553246 16953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8: No bootstrap required, opened a new log
I20260810 23:39:51.561360 16953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175e6488a8d644e6b44f3ecc775877b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46507 } }
I20260810 23:39:51.561808 16953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:51.561895 16953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 175e6488a8d644e6b44f3ecc775877b8, State: Initialized, Role: FOLLOWER
I20260810 23:39:51.562528 16953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175e6488a8d644e6b44f3ecc775877b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46507 } }
I20260810 23:39:51.562772 16953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:51.562909 16953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:51.563063 16953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:51.565721 16953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175e6488a8d644e6b44f3ecc775877b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46507 } }
I20260810 23:39:51.566334 16953 leader_election.cc:302] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 175e6488a8d644e6b44f3ecc775877b8; no voters:
I20260810 23:39:51.566926 16953 leader_election.cc:288] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:51.567068 16963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:51.567783 16963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [term 1 LEADER]: Becoming Leader. State: Replica: 175e6488a8d644e6b44f3ecc775877b8, State: Running, Role: LEADER
I20260810 23:39:51.568379 16963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175e6488a8d644e6b44f3ecc775877b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46507 } }
I20260810 23:39:51.569065 16953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:51.571273 16965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "175e6488a8d644e6b44f3ecc775877b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175e6488a8d644e6b44f3ecc775877b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46507 } } }
I20260810 23:39:51.571521 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 175e6488a8d644e6b44f3ecc775877b8. Latest consensus state: current_term: 1 leader_uuid: "175e6488a8d644e6b44f3ecc775877b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175e6488a8d644e6b44f3ecc775877b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46507 } } }
I20260810 23:39:51.571882 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:51.571766 16965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:51.573459 16976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:51.578191 16976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:51.587818 16976 catalog_manager.cc:1347] Generated new cluster ID: ccf29caad2c9400cbc04a60e9715cde5
I20260810 23:39:51.587965 16976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:51.608032 16976 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:51.608486 16976 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:51.621349 16976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 175e6488a8d644e6b44f3ecc775877b8: Generated new TSK 0
I20260810 23:39:51.622256 16976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:39:51.636039 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:46507
--builtin_ntp_servers=127.30.33.20:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:39:52.020689 16995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:52.021010 16995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:52.021131 16995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:52.028656 16995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:52.028779 16995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:52.028929 16995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:52.038676 16995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:52.040169 16995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:52.042085 16995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:52.056288 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:52.056624 17010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:52.056486 17005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:52.057150 16995 server_base.cc:1034] running on GCE node
I20260810 23:39:52.057844 16995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:52.058951 16995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:52.060174 16995 hybrid_clock.cc:648] HybridClock initialized: now 1786405192060123 us; error 48 us; skew 500 ppm
I20260810 23:39:52.060557 16995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:52.063126 16995 webserver.cc:460] Webserver started at http://127.30.33.1:43145/ using document root <none> and password file <none>
I20260810 23:39:52.063968 16995 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:52.064090 16995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:52.064431 16995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:52.067631 16995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "4827a9e5539c4ddc8efc36b403acd08d"
format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-pww0"
server_key: "d6c16504d74e0747c05fdfaf4efc039c"
server_key_iv: "c5cbe586a3d786b6669584ace0620336"
server_key_version: "encryptionkey@0"
I20260810 23:39:52.068529 16995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "4827a9e5539c4ddc8efc36b403acd08d"
format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-pww0"
server_key: "d6c16504d74e0747c05fdfaf4efc039c"
server_key_iv: "c5cbe586a3d786b6669584ace0620336"
server_key_version: "encryptionkey@0"
I20260810 23:39:52.074774 16995 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:39:52.078745 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:52.080484 16995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:52.080681 16995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "4827a9e5539c4ddc8efc36b403acd08d"
format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-pww0"
server_key: "d6c16504d74e0747c05fdfaf4efc039c"
server_key_iv: "c5cbe586a3d786b6669584ace0620336"
server_key_version: "encryptionkey@0"
I20260810 23:39:52.080940 16995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:52.087312 16995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:52.088320 16995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:52.088604 16995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:52.089625 16995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:52.091229 16995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:52.091336 16995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:52.091447 16995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:52.091552 16995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 23:39:52.124512 16995 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:37211
I20260810 23:39:52.124625 17229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:37211 every 8 connection(s)
I20260810 23:39:52.125918 16995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:52.133091 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 16995
I20260810 23:39:52.133317 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:39:52.133949 30852 external_mini_cluster.cc:1428] Setting key fceb4f2efd642d6dea75f58564d629b6
I20260810 23:39:52.134796 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:52.128671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:53009 (local address 127.30.33.62:46507)
0810 23:39:52.129156 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:52.129160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:52.130451 (+ 1291us) server_negotiation.cc:407] Connection header received
0810 23:39:52.132333 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:52.132339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:52.132411 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:52.132512 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:52.133958 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:52.133963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:52.133966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:52.133983 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:52.134042 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:52.134059 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:52.134190 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:52.134376 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 23:39:52.134453 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260810 23:39:52.135943 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:52.129273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46507 (local address 127.30.33.1:53009)
0810 23:39:52.130271 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:52.130319 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:52.132029 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:52.132797 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:52.132806 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:52.132910 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:52.133618 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:39:52.133621 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:52.133664 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:39:52.133765 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:52.134146 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:52.134157 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:52.134166 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:39:52.134452 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 23:39:52.134742 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":283,"threads_started":1}
I20260810 23:39:52.136965 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:46507
--builtin_ntp_servers=127.30.33.20:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:39:52.137743 17232 heartbeater.cc:347] Connected to a master server at 127.30.33.62:46507
I20260810 23:39:52.138059 17232 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:52.138700 17232 heartbeater.cc:511] Master 127.30.33.62:46507 requested a full tablet report, sending...
I20260810 23:39:52.141129 16889 ts_manager.cc:194] Registered new tserver with Master: 4827a9e5539c4ddc8efc36b403acd08d (127.30.33.1:37211)
W20260810 23:39:52.498713 17237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:52.499078 17237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:52.499208 17237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:52.506861 17237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:52.506979 17237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:52.507125 17237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:52.516886 17237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:52.518496 17237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:52.520383 17237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:52.534224 17245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:52.535130 17246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:52.535166 17248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:52.536820 17237 server_base.cc:1034] running on GCE node
I20260810 23:39:52.537664 17237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:52.538843 17237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:52.540079 17237 hybrid_clock.cc:648] HybridClock initialized: now 1786405192540006 us; error 68 us; skew 500 ppm
I20260810 23:39:52.540434 17237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:52.542902 17237 webserver.cc:460] Webserver started at http://127.30.33.2:42455/ using document root <none> and password file <none>
I20260810 23:39:52.543721 17237 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:52.543838 17237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:52.544170 17237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:52.547292 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "22fc5e51a901417d896274fcb802dfb0"
format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-pww0"
server_key: "9982c1ba50379632441f34bb19e4efc0"
server_key_iv: "1e3be4714ed5cbe1f2e3e1041fa65903"
server_key_version: "encryptionkey@0"
I20260810 23:39:52.548226 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "22fc5e51a901417d896274fcb802dfb0"
format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-pww0"
server_key: "9982c1ba50379632441f34bb19e4efc0"
server_key_iv: "1e3be4714ed5cbe1f2e3e1041fa65903"
server_key_version: "encryptionkey@0"
I20260810 23:39:52.554081 17237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 23:39:52.557981 17258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:52.559597 17237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:52.559796 17237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "22fc5e51a901417d896274fcb802dfb0"
format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-pww0"
server_key: "9982c1ba50379632441f34bb19e4efc0"
server_key_iv: "1e3be4714ed5cbe1f2e3e1041fa65903"
server_key_version: "encryptionkey@0"
I20260810 23:39:52.560035 17237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:52.566507 17237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:52.567492 17237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:52.567792 17237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:52.568923 17237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:52.570658 17237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:52.570776 17237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:52.570892 17237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:52.570973 17237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:52.603713 17237 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:35319
I20260810 23:39:52.603790 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:35319 every 8 connection(s)
I20260810 23:39:52.605091 17237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:52.609431 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 17237
I20260810 23:39:52.609635 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:39:52.610083 30852 external_mini_cluster.cc:1428] Setting key b3a8eb907a1dbc186e351e9133cec5ea
I20260810 23:39:52.613127 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:46507
--builtin_ntp_servers=127.30.33.20:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:39:52.614221 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:52.607872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:34273 (local address 127.30.33.62:46507)
0810 23:39:52.608092 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:52.608098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:52.609704 (+ 1606us) server_negotiation.cc:407] Connection header received
0810 23:39:52.611712 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:52.611717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:52.611787 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:52.611887 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:52.613451 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:52.613456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:52.613459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:52.613479 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:52.613516 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:52.613529 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:52.613659 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:52.613865 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 23:39:52.613940 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:39:52.615274 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:52.608513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46507 (local address 127.30.33.2:34273)
0810 23:39:52.609527 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:52.609575 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:52.611389 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:52.612151 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:52.612160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:52.612270 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:52.613020 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:39:52.613023 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:52.613069 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:39:52.613183 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:52.613647 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:52.613659 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:52.613669 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:39:52.613949 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 23:39:52.614252 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":282,"threads_started":1}
I20260810 23:39:52.616822 17494 heartbeater.cc:347] Connected to a master server at 127.30.33.62:46507
I20260810 23:39:52.617153 17494 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:52.617790 17494 heartbeater.cc:511] Master 127.30.33.62:46507 requested a full tablet report, sending...
I20260810 23:39:52.619305 16889 ts_manager.cc:194] Registered new tserver with Master: 22fc5e51a901417d896274fcb802dfb0 (127.30.33.2:35319)
W20260810 23:39:52.966081 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:52.966398 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:52.966539 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:52.974231 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:52.974356 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:52.974498 17501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:52.983875 17501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:52.985340 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:52.987229 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:53.003693 17511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:53.003695 17513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:53.003732 17509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:53.004729 17501 server_base.cc:1034] running on GCE node
I20260810 23:39:53.005369 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:53.006367 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:53.007576 17501 hybrid_clock.cc:648] HybridClock initialized: now 1786405193007521 us; error 51 us; skew 500 ppm
I20260810 23:39:53.007928 17501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:39:53.010358 17501 webserver.cc:460] Webserver started at http://127.30.33.3:36905/ using document root <none> and password file <none>
I20260810 23:39:53.011220 17501 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:53.011353 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:53.011734 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:53.014808 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "37720f103af5417b884798fc4f4b5e49"
format_stamp: "Formatted at 2026-08-10 23:39:53 on dist-test-slave-pww0"
server_key: "2b863030cabe4aa3095e3bf620780592"
server_key_iv: "928ee7207a98cced620da3c277d092eb"
server_key_version: "encryptionkey@0"
I20260810 23:39:53.015717 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "37720f103af5417b884798fc4f4b5e49"
format_stamp: "Formatted at 2026-08-10 23:39:53 on dist-test-slave-pww0"
server_key: "2b863030cabe4aa3095e3bf620780592"
server_key_iv: "928ee7207a98cced620da3c277d092eb"
server_key_version: "encryptionkey@0"
I20260810 23:39:53.021477 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:39:53.025244 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:53.026666 17501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:39:53.026847 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "37720f103af5417b884798fc4f4b5e49"
format_stamp: "Formatted at 2026-08-10 23:39:53 on dist-test-slave-pww0"
server_key: "2b863030cabe4aa3095e3bf620780592"
server_key_iv: "928ee7207a98cced620da3c277d092eb"
server_key_version: "encryptionkey@0"
I20260810 23:39:53.027070 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:53.032925 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:53.033803 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:53.034072 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:53.035065 17501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:53.036717 17501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:53.036820 17501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:53.036940 17501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:53.037019 17501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:53.068815 17501 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:45109
I20260810 23:39:53.068884 17738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:45109 every 8 connection(s)
I20260810 23:39:53.070338 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:53.074522 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 17501
I20260810 23:39:53.074744 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:39:53.075239 30852 external_mini_cluster.cc:1428] Setting key 01ac1a1ae0946089237411dc0a522fb8
I20260810 23:39:53.079484 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:53.073112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:47335 (local address 127.30.33.62:46507)
0810 23:39:53.073330 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:53.073336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:53.074970 (+ 1634us) server_negotiation.cc:407] Connection header received
0810 23:39:53.077070 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:53.077076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:53.077152 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:53.077255 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:53.078697 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:53.078702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:53.078705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:53.078723 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:53.078759 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:53.078772 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:53.078878 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:53.079080 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 23:39:53.079157 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:39:53.080564 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:53.073747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46507 (local address 127.30.33.3:47335)
0810 23:39:53.074781 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:53.074835 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:53.076787 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:53.077531 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:53.077540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:53.077651 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:53.078375 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:39:53.078378 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:53.078424 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:39:53.078524 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:53.078862 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:53.078874 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:53.078883 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:39:53.079168 (+ 285us) client_negotiation.cc:240] Negotiation successful
0810 23:39:53.079509 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":290,"threads_started":1}
I20260810 23:39:53.081974 17739 heartbeater.cc:347] Connected to a master server at 127.30.33.62:46507
I20260810 23:39:53.082315 17739 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:53.082962 17739 heartbeater.cc:511] Master 127.30.33.62:46507 requested a full tablet report, sending...
I20260810 23:39:53.084576 16889 ts_manager.cc:194] Registered new tserver with Master: 37720f103af5417b884798fc4f4b5e49 (127.30.33.3:45109)
I20260810 23:39:53.091123 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:39:53.097625 17234 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:39:53.096035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.30.33.62:46507)
0810 23:39:53.096234 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:53.096238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:53.096727 (+ 489us) server_negotiation.cc:407] Connection header received
0810 23:39:53.096877 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:53.096882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:53.096938 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:53.097027 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:53.097329 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56238: BlockingRecv error: recv got EOF from 127.0.0.1:56238 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:39:53.098127 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 16995
I20260810 23:39:53.101898 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:53.099243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.30.33.1:37211)
0810 23:39:53.099845 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:53.099853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:53.099881 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:39:53.099944 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:53.099950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:53.100164 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:53.100309 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:53.101202 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:53.101208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:53.101221 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:53.101239 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:53.101281 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:53.101296 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:53.101432 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:53.101491 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:39:53.101577 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":326,"threads_started":1}
I20260810 23:39:53.103233 17224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:53.188328 17232 heartbeater.cc:503] Master 127.30.33.62:46507 was elected leader, sending a full tablet report...
I20260810 23:39:53.189543 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 16995
W20260810 23:39:53.216414 16869 connection.cc:535] server connection from 127.30.33.1:53009 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:39:53.216604 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 17237
I20260810 23:39:53.220255 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:53.217518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.30.33.2:35319)
0810 23:39:53.218077 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:53.218084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:53.218112 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:39:53.218179 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:53.218186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:53.218419 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:53.218552 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:53.219518 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:53.219529 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:53.219545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:53.219569 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:53.219616 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:53.219632 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:53.219750 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:53.219852 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 23:39:53.219956 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
I20260810 23:39:53.221593 17488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:53.306059 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 17237
I20260810 23:39:53.330310 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 17501
I20260810 23:39:53.334444 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:53.331373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.30.33.3:45109)
0810 23:39:53.332141 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:53.332148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:53.332181 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:39:53.332247 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:53.332253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:53.332478 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:53.332617 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:53.333663 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:53.333671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:53.333685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:39:53.333706 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:53.333753 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:39:53.333769 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:53.333912 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:53.333982 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:39:53.334102 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":403,"threads_started":1}
I20260810 23:39:53.335843 17733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:53.417115 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 17501
I20260810 23:39:53.441586 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 16831
I20260810 23:39:53.442958 16941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:53.554354 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 16831
2026-08-10T23:39:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2571 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:39:53 dist-test-slave-pww0 krb5kdc[17770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:39:53 dist-test-slave-pww0 krb5kdc[17770](info): set up 2 sockets
Aug 10 23:39:53 dist-test-slave-pww0 krb5kdc[17770](info): commencing operation
krb5kdc: starting...
W20260810 23:39:54.859630 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.002s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:39:55 dist-test-slave-pww0 krb5kdc[17770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:39:55.190907 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:43295
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:55.550678 17801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:55.551021 17801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:55.551129 17801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:55.558969 17801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:39:55.559108 17801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:55.559157 17801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:55.559242 17801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:39:55.559296 17801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:39:55.570315 17801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:43295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:55.571796 17801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:55.573231 17801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:55.585760 17810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:55.585673 17811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:55.585718 17814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:55.587044 17801 server_base.cc:1034] running on GCE node
I20260810 23:39:55.587985 17801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:55.589560 17801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:55.590797 17801 hybrid_clock.cc:648] HybridClock initialized: now 1786405195590741 us; error 47 us; skew 500 ppm
I20260810 23:39:55.591153 17801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:55 dist-test-slave-pww0 krb5kdc[17770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405195, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:55.594414 17801 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:39:55.596127 17801 webserver.cc:460] Webserver started at http://127.30.33.62:36813/ using document root <none> and password file <none>
I20260810 23:39:55.596853 17801 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:55.596977 17801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:55.597299 17801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:55.600512 17801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "168290d7c42c4d3d97c2f9fbef0269c7"
format_stamp: "Formatted at 2026-08-10 23:39:55 on dist-test-slave-pww0"
server_key: "1a95a1f3ced4bd36ffe4424c99fc7de5"
server_key_iv: "fc39b6faac6269901ee2f3fad70390a4"
server_key_version: "encryptionkey@0"
I20260810 23:39:55.601316 17801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "168290d7c42c4d3d97c2f9fbef0269c7"
format_stamp: "Formatted at 2026-08-10 23:39:55 on dist-test-slave-pww0"
server_key: "1a95a1f3ced4bd36ffe4424c99fc7de5"
server_key_iv: "fc39b6faac6269901ee2f3fad70390a4"
server_key_version: "encryptionkey@0"
I20260810 23:39:55.607179 17801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:39:55.610939 17824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:55.612444 17801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:39:55.612625 17801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "168290d7c42c4d3d97c2f9fbef0269c7"
format_stamp: "Formatted at 2026-08-10 23:39:55 on dist-test-slave-pww0"
server_key: "1a95a1f3ced4bd36ffe4424c99fc7de5"
server_key_iv: "fc39b6faac6269901ee2f3fad70390a4"
server_key_version: "encryptionkey@0"
I20260810 23:39:55.612840 17801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:55.628022 17801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:55.632197 17801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:55.632481 17801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:55.651443 17801 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:43295
I20260810 23:39:55.651527 17925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:43295 every 8 connection(s)
I20260810 23:39:55.652978 17801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:39:55.653402 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 17801
I20260810 23:39:55.653622 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:39:55.654495 30852 external_mini_cluster.cc:1428] Setting key 30bf8bd9e4fe971cd5ce6866b3d657cf
I20260810 23:39:55.658074 17926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:39:55 dist-test-slave-pww0 krb5kdc[17770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:55.670369 17926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7: Bootstrap starting.
I20260810 23:39:55.674206 17926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:55.675343 17926 log.cc:824] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:55.679035 17926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7: No bootstrap required, opened a new log
I20260810 23:39:55.679188 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:55.656866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.30.33.62:43295)
0810 23:39:55.657674 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:55.657687 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:55.657737 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:39:55.659543 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:55.659570 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:55.660009 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:55.660415 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:55.662086 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:55.663613 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:55.664980 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:55.665714 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:55.665907 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:55.670723 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:55.670735 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:55.670753 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:55.670812 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:55.674326 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:55.675201 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:55.675209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:55.675217 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:55.675430 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:55.676079 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:55.676084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:55.676088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:55.676923 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:55.677202 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:55.677652 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 23:39:55.678061 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":360,"threads_started":1}
I20260810 23:39:55.686908 17926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43295 } }
I20260810 23:39:55.687364 17926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:55.687521 17926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168290d7c42c4d3d97c2f9fbef0269c7, State: Initialized, Role: FOLLOWER
I20260810 23:39:55.688140 17926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43295 } }
I20260810 23:39:55.688376 17926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:39:55.688479 17926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:39:55.688619 17926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:55.691116 17926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43295 } }
I20260810 23:39:55.691696 17926 leader_election.cc:302] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 168290d7c42c4d3d97c2f9fbef0269c7; no voters:
I20260810 23:39:55.692293 17926 leader_election.cc:288] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:39:55.692422 17933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:55.693118 17933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [term 1 LEADER]: Becoming Leader. State: Replica: 168290d7c42c4d3d97c2f9fbef0269c7, State: Running, Role: LEADER
I20260810 23:39:55.693657 17933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43295 } }
I20260810 23:39:55.694494 17926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:39:55.696491 17934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43295 } } }
I20260810 23:39:55.696899 17934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:55.699896 17943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:39:55.699682 17936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 168290d7c42c4d3d97c2f9fbef0269c7. Latest consensus state: current_term: 1 leader_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168290d7c42c4d3d97c2f9fbef0269c7" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43295 } } }
I20260810 23:39:55.700057 17936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:39:55.705658 17943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:39:55.715793 17943 catalog_manager.cc:1347] Generated new cluster ID: f61154e807b44777950f41065433075c
I20260810 23:39:55.715927 17943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:39:55.729338 17943 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:39:55.730801 17943 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:39:55.745306 17943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7: Generated new TSK 0
I20260810 23:39:55.746289 17943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:39:55.824992 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43295
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:56.192405 17968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:56.192713 17968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:56.192847 17968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:56.200771 17968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:56.200910 17968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:56.201072 17968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:39:56.211491 17968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:56.212908 17968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:56.214298 17968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:56.228082 17982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:56.228056 17986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:56.228017 17988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:56.228752 17968 server_base.cc:1034] running on GCE node
I20260810 23:39:56.229408 17968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:56.230436 17968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:56.231638 17968 hybrid_clock.cc:648] HybridClock initialized: now 1786405196231579 us; error 50 us; skew 500 ppm
I20260810 23:39:56.231966 17968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:56 dist-test-slave-pww0 krb5kdc[17770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405196, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:56.235327 17968 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:39:56.236901 17968 webserver.cc:460] Webserver started at http://127.30.33.1:39523/ using document root <none> and password file <none>
I20260810 23:39:56.237635 17968 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:56.237764 17968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:56.238071 17968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:56.241217 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "01e07de2df5e407299e313319f730abd"
format_stamp: "Formatted at 2026-08-10 23:39:56 on dist-test-slave-pww0"
server_key: "bb3aa6bb0a1dafef7f1227acfa6b6e68"
server_key_iv: "265292a154b04adb6bc5a3923c614128"
server_key_version: "encryptionkey@0"
I20260810 23:39:56.241994 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "01e07de2df5e407299e313319f730abd"
format_stamp: "Formatted at 2026-08-10 23:39:56 on dist-test-slave-pww0"
server_key: "bb3aa6bb0a1dafef7f1227acfa6b6e68"
server_key_iv: "265292a154b04adb6bc5a3923c614128"
server_key_version: "encryptionkey@0"
I20260810 23:39:56.247983 17968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:39:56.251883 18007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:56.253394 17968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:39:56.253584 17968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "01e07de2df5e407299e313319f730abd"
format_stamp: "Formatted at 2026-08-10 23:39:56 on dist-test-slave-pww0"
server_key: "bb3aa6bb0a1dafef7f1227acfa6b6e68"
server_key_iv: "265292a154b04adb6bc5a3923c614128"
server_key_version: "encryptionkey@0"
I20260810 23:39:56.253819 17968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:56.283592 17968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:56.288168 17968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:56.288483 17968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:56.289520 17968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:56.291040 17968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:56.291122 17968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:56.291254 17968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:56.291311 17968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:56.324607 17968 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:42639
I20260810 23:39:56.324663 18202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:42639 every 8 connection(s)
I20260810 23:39:56.326004 17968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:39:56.327515 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 17968
I20260810 23:39:56.327766 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:39:56.328636 30852 external_mini_cluster.cc:1428] Setting key 91108c91203785c555380d86d0414442
Aug 10 23:39:56 dist-test-slave-pww0 krb5kdc[17770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405196, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:56.351763 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:56.329221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:47543 (local address 127.30.33.62:43295)
0810 23:39:56.329773 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:56.329779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:56.331054 (+ 1275us) server_negotiation.cc:407] Connection header received
0810 23:39:56.333843 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:56.333848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:56.333928 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:56.334019 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:56.336354 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.337142 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.339551 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.340016 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.340153 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:56.344877 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:56.344884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:56.344887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:56.344932 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:56.348203 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:56.349113 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:56.349120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:56.349123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:56.349322 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:56.349787 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:56.349791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:56.349794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:56.350143 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:56.350302 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:56.351110 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 23:39:56.351363 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
I20260810 23:39:56.352646 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:56.329911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43295 (local address 127.30.33.1:47543)
0810 23:39:56.330897 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:56.330949 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:56.333440 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:56.334368 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:56.334381 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:56.335174 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:56.336097 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:56.336128 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.337353 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.337359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:56.339280 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:56.339300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.340311 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.340319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:56.340640 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:56.341483 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:56.341515 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:56.344519 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:56.348401 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:56.348410 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:56.348426 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:56.348940 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:56.349437 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:56.349440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:56.349443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:56.349649 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:56.350275 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:56.350282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:56.350655 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 23:39:56.350910 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 23:39:56.351380 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":430,"threads_started":1}
I20260810 23:39:56.354324 18203 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43295
I20260810 23:39:56.354650 18203 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:56.355499 18203 heartbeater.cc:511] Master 127.30.33.62:43295 requested a full tablet report, sending...
I20260810 23:39:56.357906 17854 ts_manager.cc:194] Registered new tserver with Master: 01e07de2df5e407299e313319f730abd (127.30.33.1:42639)
I20260810 23:39:56.359733 17854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:47543
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:39:56.403371 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43295
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:56.769855 18212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:56.770187 18212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:56.770355 18212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:56.777815 18212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:56.777961 18212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:56.778120 18212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:39:56.788275 18212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:56.789700 18212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:56.791145 18212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:39:56.805115 18222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:56.805176 18219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:56.805411 18218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:56.806103 18212 server_base.cc:1034] running on GCE node
I20260810 23:39:56.806788 18212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:56.807914 18212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:56.809130 18212 hybrid_clock.cc:648] HybridClock initialized: now 1786405196809070 us; error 55 us; skew 500 ppm
I20260810 23:39:56.809481 18212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:56 dist-test-slave-pww0 krb5kdc[17770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405196, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:56.812930 18212 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:39:56.814603 18212 webserver.cc:460] Webserver started at http://127.30.33.2:36313/ using document root <none> and password file <none>
I20260810 23:39:56.815389 18212 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:56.815565 18212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:56.815871 18212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:56.819139 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "e681426e752b423197b2a936921a6785"
format_stamp: "Formatted at 2026-08-10 23:39:56 on dist-test-slave-pww0"
server_key: "ccb84a6dd681abca04e3261160e07d2e"
server_key_iv: "b2bc0ae13896efc8d0e68b21d23d3065"
server_key_version: "encryptionkey@0"
I20260810 23:39:56.819988 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "e681426e752b423197b2a936921a6785"
format_stamp: "Formatted at 2026-08-10 23:39:56 on dist-test-slave-pww0"
server_key: "ccb84a6dd681abca04e3261160e07d2e"
server_key_iv: "b2bc0ae13896efc8d0e68b21d23d3065"
server_key_version: "encryptionkey@0"
I20260810 23:39:56.825842 18212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:39:56.829746 18234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:56.831271 18212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:39:56.831439 18212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "e681426e752b423197b2a936921a6785"
format_stamp: "Formatted at 2026-08-10 23:39:56 on dist-test-slave-pww0"
server_key: "ccb84a6dd681abca04e3261160e07d2e"
server_key_iv: "b2bc0ae13896efc8d0e68b21d23d3065"
server_key_version: "encryptionkey@0"
I20260810 23:39:56.831718 18212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:56.848405 18212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:56.852999 18212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:56.853303 18212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:56.854367 18212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:56.855947 18212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:56.856026 18212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:56.856145 18212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:56.856196 18212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:56.889251 18212 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:41151
I20260810 23:39:56.889305 18423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:41151 every 8 connection(s)
I20260810 23:39:56.890708 18212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:39:56.896943 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 18212
I20260810 23:39:56.897146 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:39:56.897722 30852 external_mini_cluster.cc:1428] Setting key e6926047fcab81e02ec90c3b4aca5704
Aug 10 23:39:56 dist-test-slave-pww0 krb5kdc[17770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405196, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:56.913789 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:56.893519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:35785 (local address 127.30.33.62:43295)
0810 23:39:56.894009 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:56.894013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:56.895022 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 23:39:56.897187 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:56.897195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:56.897299 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:56.897439 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:56.899351 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.900375 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.902183 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.902796 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.902927 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:56.906563 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:56.906570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:56.906574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:56.906618 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:56.909898 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:56.910960 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:56.910966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:56.910970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:56.911153 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:56.911786 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:56.911791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:56.911794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:56.912178 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:56.912337 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:56.913150 (+ 813us) server_negotiation.cc:299] Negotiation successful
0810 23:39:56.913406 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260810 23:39:56.915135 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:56.893990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43295 (local address 127.30.33.2:35785)
0810 23:39:56.894840 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:56.894900 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:56.896835 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:56.897682 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:56.897692 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:56.898299 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:56.899155 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:56.899185 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.900545 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.900550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:56.902016 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:56.902031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:56.903025 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:56.903030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:56.903257 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:56.903959 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:56.903983 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:56.906287 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:56.910120 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:56.910228 (+ 108us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:56.910257 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:56.910807 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:56.911329 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:56.911334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:56.911336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:56.911652 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:56.912345 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:56.912353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:56.912753 (+ 400us) client_negotiation.cc:769] Sending connection context
0810 23:39:56.913011 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 23:39:56.913484 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":322,"threads_started":1}
I20260810 23:39:56.916850 18425 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43295
I20260810 23:39:56.917214 18425 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:56.918257 18425 heartbeater.cc:511] Master 127.30.33.62:43295 requested a full tablet report, sending...
I20260810 23:39:56.920032 17854 ts_manager.cc:194] Registered new tserver with Master: e681426e752b423197b2a936921a6785 (127.30.33.2:41151)
I20260810 23:39:56.920933 17854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:35785
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:39:56.971694 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43295
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:39:57.326969 18433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:39:57.327385 18433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:39:57.327575 18433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:39:57.336382 18433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:39:57.336602 18433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:39:57.336740 18433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:39:57.347872 18433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:39:57.349320 18433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:39:57.350736 18433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:39:57.363009 18203 heartbeater.cc:503] Master 127.30.33.62:43295 was elected leader, sending a full tablet report...
W20260810 23:39:57.365124 18445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:57.365453 18443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:39:57.365232 18442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:39:57.365762 18433 server_base.cc:1034] running on GCE node
I20260810 23:39:57.366613 18433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:39:57.367668 18433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:39:57.368901 18433 hybrid_clock.cc:648] HybridClock initialized: now 1786405197368841 us; error 47 us; skew 500 ppm
I20260810 23:39:57.369263 18433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:39:57 dist-test-slave-pww0 krb5kdc[17770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405197, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:39:57.372797 18433 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:39:57.374369 18433 webserver.cc:460] Webserver started at http://127.30.33.3:44753/ using document root <none> and password file <none>
I20260810 23:39:57.375118 18433 fs_manager.cc:359] Metadata directory not provided
I20260810 23:39:57.375232 18433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:39:57.375589 18433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:39:57.378722 18433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "909dcebcc146402282b1fd30a0a31183"
format_stamp: "Formatted at 2026-08-10 23:39:57 on dist-test-slave-pww0"
server_key: "73f2a6ada3a9da4d82edc9c46fc18b95"
server_key_iv: "c1bdef9e0d43c3f44309a44aba5ed965"
server_key_version: "encryptionkey@0"
I20260810 23:39:57.379709 18433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "909dcebcc146402282b1fd30a0a31183"
format_stamp: "Formatted at 2026-08-10 23:39:57 on dist-test-slave-pww0"
server_key: "73f2a6ada3a9da4d82edc9c46fc18b95"
server_key_iv: "c1bdef9e0d43c3f44309a44aba5ed965"
server_key_version: "encryptionkey@0"
I20260810 23:39:57.385659 18433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:39:57.389585 18458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:57.391185 18433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:39:57.391367 18433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "909dcebcc146402282b1fd30a0a31183"
format_stamp: "Formatted at 2026-08-10 23:39:57 on dist-test-slave-pww0"
server_key: "73f2a6ada3a9da4d82edc9c46fc18b95"
server_key_iv: "c1bdef9e0d43c3f44309a44aba5ed965"
server_key_version: "encryptionkey@0"
I20260810 23:39:57.391649 18433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:39:57.404575 18433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:39:57.409052 18433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:39:57.409355 18433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:39:57.410416 18433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:39:57.411914 18433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:39:57.411990 18433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:57.412117 18433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:39:57.412168 18433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:39:57.444038 18433 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:40027
I20260810 23:39:57.444087 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:40027 every 8 connection(s)
I20260810 23:39:57.445472 18433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:39:57.455282 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 18433
I20260810 23:39:57.455518 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:39:57.455972 30852 external_mini_cluster.cc:1428] Setting key 59d88c878983f067a8c7e3ee45eba1bf
Aug 10 23:39:57 dist-test-slave-pww0 krb5kdc[17770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405197, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:39:57.467048 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.448293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:43047 (local address 127.30.33.62:43295)
0810 23:39:57.448750 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.448754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.449789 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 23:39:57.451775 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.451780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.451895 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.451997 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:57.453734 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.454528 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.456296 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.456875 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.457003 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.460968 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:57.460974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:57.460978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:57.461016 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:57.463667 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:57.464514 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:57.464520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:57.464522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:57.464659 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:57.465133 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:57.465137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:57.465139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:57.465447 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:57.465601 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.466409 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.466659 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":225,"threads_started":1}
I20260810 23:39:57.468127 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.448783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43295 (local address 127.30.33.3:43047)
0810 23:39:57.449590 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.449632 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.451398 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.452287 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.452298 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.452894 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:39:57.453537 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.453556 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.454695 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.454699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.456131 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.456153 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.457126 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.457134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.457439 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.458121 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:39:57.458147 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:39:57.460709 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:39:57.463857 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:57.463865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:57.463883 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:57.464371 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:57.464787 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:39:57.464791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:39:57.464794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:39:57.465003 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:39:57.465590 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:39:57.465599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:39:57.465991 (+ 392us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.466256 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.466884 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":338,"threads_started":1}
I20260810 23:39:57.469590 18678 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43295
I20260810 23:39:57.469954 18678 heartbeater.cc:464] Registering TS with master...
I20260810 23:39:57.470891 18678 heartbeater.cc:511] Master 127.30.33.62:43295 requested a full tablet report, sending...
I20260810 23:39:57.472536 17854 ts_manager.cc:194] Registered new tserver with Master: 909dcebcc146402282b1fd30a0a31183 (127.30.33.3:40027)
I20260810 23:39:57.473361 17854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:43047
I20260810 23:39:57.484510 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:39:57.503082 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.489342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.30.33.62:43295)
0810 23:39:57.489545 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.489550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.490016 (+ 466us) server_negotiation.cc:407] Connection header received
0810 23:39:57.490207 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.490211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.490318 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.490410 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:57.491960 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.492756 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.494026 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.494532 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.494650 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.497132 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:57.497138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:57.497141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:57.497182 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:57.500063 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:57.500820 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:57.500825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:57.500827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:57.501008 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:57.501498 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:57.501503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:57.501505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:57.501972 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:57.502217 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.502639 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.502836 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:39:57.507534 17854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:39:57.514822 17854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:39:57.561285 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.531057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.30.33.2:41151)
0810 23:39:57.551254 (+ 20197us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.551266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.552504 (+ 1238us) server_negotiation.cc:407] Connection header received
0810 23:39:57.552604 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.552609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.552782 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.552990 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:57.553793 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.555383 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.558275 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.560045 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.560188 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.560422 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.560755 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.560934 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19971,"thread_start_us":19803,"threads_started":1}
I20260810 23:39:57.561363 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.530451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41151 (local address 127.0.0.1:43000)
0810 23:39:57.552198 (+ 21747us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.552244 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.552401 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.553027 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.553034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.553074 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:57.553497 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.553511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.555554 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.555560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.557831 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.557847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.560283 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.560289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.560493 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.560537 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.560668 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.560893 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21637,"mutex_wait_us":57,"thread_start_us":21015,"threads_started":1}
I20260810 23:39:57.562227 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.528078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:40027 (local address 127.0.0.1:36906)
0810 23:39:57.544824 (+ 16746us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.544881 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.545131 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.554380 (+ 9249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.554385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.554411 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:57.554791 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.554803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.556887 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.556893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.559001 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.559016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.561175 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.561183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.561419 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.561523 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.561614 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.561803 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16437,"mutex_wait_us":91,"thread_start_us":16278,"threads_started":1}
I20260810 23:39:57.562592 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.528444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42639 (local address 127.0.0.1:55478)
0810 23:39:57.551824 (+ 23380us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.551855 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.552052 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.553726 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.553730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.553756 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:57.554175 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.554188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.555910 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.555916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.557893 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.557909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.561196 (+ 3287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.561203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.561482 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.561539 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.561614 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.562253 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23213,"mutex_wait_us":515,"thread_start_us":23477,"threads_started":1}
I20260810 23:39:57.563074 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.528320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.30.33.3:40027)
0810 23:39:57.553750 (+ 25430us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.553758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.553790 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:39:57.553887 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.553895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.554132 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.554988 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:57.555261 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.556716 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.559188 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.560609 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.560762 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.560985 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.562485 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.562678 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25242,"thread_start_us":25065,"threads_started":1}
I20260810 23:39:57.563764 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.528660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.30.33.1:42639)
0810 23:39:57.544073 (+ 15413us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.544086 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.552254 (+ 8168us) server_negotiation.cc:407] Connection header received
0810 23:39:57.552404 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.552414 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.552770 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.552989 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:57.554582 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.555756 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.558059 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.560619 (+ 2560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.560744 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.560987 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.561755 (+ 768us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.563491 (+ 1736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15150,"thread_start_us":15439,"threads_started":1}
I20260810 23:39:57.566156 18310 tablet_service.cc:1467] Processing CreateTablet for tablet bf44dccf8923478c8b4c3a9475344ac1 (DEFAULT_TABLE table=test-table [id=c3fd1bfc15894cfe898088614e90edf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:57.567890 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf44dccf8923478c8b4c3a9475344ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:57.567821 18084 tablet_service.cc:1467] Processing CreateTablet for tablet bf44dccf8923478c8b4c3a9475344ac1 (DEFAULT_TABLE table=test-table [id=c3fd1bfc15894cfe898088614e90edf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:57.569444 18084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf44dccf8923478c8b4c3a9475344ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:57.570364 18552 tablet_service.cc:1467] Processing CreateTablet for tablet bf44dccf8923478c8b4c3a9475344ac1 (DEFAULT_TABLE table=test-table [id=c3fd1bfc15894cfe898088614e90edf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:39:57.571743 18552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf44dccf8923478c8b4c3a9475344ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:39:57.583261 18718 tablet_bootstrap.cc:492] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Bootstrap starting.
I20260810 23:39:57.585942 18721 tablet_bootstrap.cc:492] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Bootstrap starting.
I20260810 23:39:57.588480 18718 tablet_bootstrap.cc:654] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:57.589565 18719 tablet_bootstrap.cc:492] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Bootstrap starting.
I20260810 23:39:57.589849 18721 tablet_bootstrap.cc:654] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:57.590070 18718 log.cc:824] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:57.591253 18721 log.cc:824] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:57.593215 18719 tablet_bootstrap.cc:654] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Neither blocks nor log segments found. Creating new log.
I20260810 23:39:57.594267 18719 log.cc:824] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Log is configured to *not* fsync() on all Append() calls
I20260810 23:39:57.594363 18718 tablet_bootstrap.cc:492] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: No bootstrap required, opened a new log
I20260810 23:39:57.594363 18721 tablet_bootstrap.cc:492] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: No bootstrap required, opened a new log
I20260810 23:39:57.594688 18721 ts_tablet_manager.cc:1397] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 23:39:57.594684 18718 ts_tablet_manager.cc:1397] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260810 23:39:57.603564 18719 tablet_bootstrap.cc:492] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: No bootstrap required, opened a new log
I20260810 23:39:57.603892 18719 ts_tablet_manager.cc:1397] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260810 23:39:57.605585 18718 raft_consensus.cc:348] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.606123 18718 raft_consensus.cc:374] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:57.606230 18718 raft_consensus.cc:729] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909dcebcc146402282b1fd30a0a31183, State: Initialized, Role: FOLLOWER
I20260810 23:39:57.606102 18721 raft_consensus.cc:348] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.606590 18721 raft_consensus.cc:374] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:57.606704 18721 raft_consensus.cc:729] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e07de2df5e407299e313319f730abd, State: Initialized, Role: FOLLOWER
I20260810 23:39:57.607048 18718 consensus_queue.cc:260] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.607342 18721 consensus_queue.cc:260] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.608399 18678 heartbeater.cc:503] Master 127.30.33.62:43295 was elected leader, sending a full tablet report...
I20260810 23:39:57.609140 18718 ts_tablet_manager.cc:1428] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260810 23:39:57.610949 18721 ts_tablet_manager.cc:1428] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260810 23:39:57.613391 18719 raft_consensus.cc:348] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.613847 18719 raft_consensus.cc:374] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:39:57.613945 18719 raft_consensus.cc:729] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e681426e752b423197b2a936921a6785, State: Initialized, Role: FOLLOWER
I20260810 23:39:57.614645 18719 consensus_queue.cc:260] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.615777 18425 heartbeater.cc:503] Master 127.30.33.62:43295 was elected leader, sending a full tablet report...
I20260810 23:39:57.616466 18719 ts_tablet_manager.cc:1428] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260810 23:39:57.642870 18426 tablet.cc:2369] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:39:57.697196 18679 tablet.cc:2369] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:39:57.801929 18730 raft_consensus.cc:482] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:57.802243 18730 raft_consensus.cc:504] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.804155 18730 leader_election.cc:288] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909dcebcc146402282b1fd30a0a31183 (127.30.33.3:40027), e681426e752b423197b2a936921a6785 (127.30.33.2:41151)
I20260810 23:39:57.805006 18731 raft_consensus.cc:482] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:39:57.805286 18731 raft_consensus.cc:504] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.808177 18731 leader_election.cc:288] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909dcebcc146402282b1fd30a0a31183 (127.30.33.3:40027), 01e07de2df5e407299e313319f730abd (127.30.33.1:42639)
I20260810 23:39:57.813777 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.804638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:34227 (local address 127.30.33.3:40027)
0810 23:39:57.804874 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.804881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.805844 (+ 963us) server_negotiation.cc:407] Connection header received
0810 23:39:57.806185 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.806191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.806292 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.806364 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:57.807257 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.808383 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.811033 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.812429 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.812538 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.812656 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.813304 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.813469 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 23:39:57.814679 18733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.804824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:40027 (local address 127.30.33.1:34227)
0810 23:39:57.805703 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.805724 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.805984 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.806576 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.806581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.806609 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:57.807089 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.807102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.808567 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.808572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.810818 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.810838 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.812877 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.812883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.813080 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.813131 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.814065 (+ 934us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.814255 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":3,"thread_start_us":469,"threads_started":1}
I20260810 23:39:57.817094 18589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf44dccf8923478c8b4c3a9475344ac1" candidate_uuid: "01e07de2df5e407299e313319f730abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909dcebcc146402282b1fd30a0a31183" is_pre_election: true
I20260810 23:39:57.817755 18589 raft_consensus.cc:2454] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e07de2df5e407299e313319f730abd in term 0.
I20260810 23:39:57.818565 18012 leader_election.cc:302] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e07de2df5e407299e313319f730abd, 909dcebcc146402282b1fd30a0a31183; no voters:
I20260810 23:39:57.819033 18730 raft_consensus.cc:2781] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:39:57.819202 18730 raft_consensus.cc:482] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:39:57.819339 18730 raft_consensus.cc:3037] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:57.820407 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.804671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:42685 (local address 127.30.33.2:41151)
0810 23:39:57.804883 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.804890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.806157 (+ 1267us) server_negotiation.cc:407] Connection header received
0810 23:39:57.806334 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.806339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.806404 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.806484 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:57.807124 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.808374 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.817234 (+ 8860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.818617 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.818735 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.819178 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.819956 (+ 778us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.820112 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":325,"server-negotiator.queue_time_us":54}
I20260810 23:39:57.821285 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.805023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41151 (local address 127.30.33.1:42685)
0810 23:39:57.806025 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.806042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.806197 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.806518 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.806522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.806548 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:57.806941 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.806950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.814961 (+ 8011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.814971 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.817043 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.817059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.819562 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.819570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.819761 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.819796 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.820740 (+ 944us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.820907 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":688,"threads_started":1}
I20260810 23:39:57.822858 18730 raft_consensus.cc:504] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.822794 18344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf44dccf8923478c8b4c3a9475344ac1" candidate_uuid: "01e07de2df5e407299e313319f730abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681426e752b423197b2a936921a6785" is_pre_election: true
I20260810 23:39:57.823313 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.809150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:41779 (local address 127.30.33.1:42639)
0810 23:39:57.809330 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.809335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.815877 (+ 6542us) server_negotiation.cc:407] Connection header received
0810 23:39:57.816194 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.816200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.816291 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.816351 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:57.817161 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.818182 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.820832 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.822185 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.822316 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.822412 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.822956 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.823097 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:39:57.823354 18344 raft_consensus.cc:2454] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e07de2df5e407299e313319f730abd in term 0.
I20260810 23:39:57.823833 18730 leader_election.cc:288] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [CANDIDATE]: Term 1 election: Requested vote from peers 909dcebcc146402282b1fd30a0a31183 (127.30.33.3:40027), e681426e752b423197b2a936921a6785 (127.30.33.2:41151)
I20260810 23:39:57.823998 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.807674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42639 (local address 127.30.33.2:41779)
0810 23:39:57.815772 (+ 8098us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.815793 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.816025 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.816555 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.816559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.816587 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:57.816970 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.816983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.818419 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.818424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.820648 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.820662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.822576 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.822583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.822749 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.822787 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.823490 (+ 703us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.823632 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7955,"mutex_wait_us":1,"thread_start_us":7766,"threads_started":1}
I20260810 23:39:57.824019 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.808793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:49083 (local address 127.30.33.3:40027)
0810 23:39:57.814019 (+ 5226us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.814024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.816318 (+ 2294us) server_negotiation.cc:407] Connection header received
0810 23:39:57.816626 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.816631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.816726 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.816804 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:39:57.817877 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.818960 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.821556 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.822750 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.822880 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.822981 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.823572 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.823705 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5133,"thread_start_us":7264,"threads_started":1}
I20260810 23:39:57.824350 18589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf44dccf8923478c8b4c3a9475344ac1" candidate_uuid: "01e07de2df5e407299e313319f730abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909dcebcc146402282b1fd30a0a31183"
I20260810 23:39:57.824566 18589 raft_consensus.cc:3037] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:57.824677 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.807871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:40027 (local address 127.30.33.2:49083)
0810 23:39:57.816220 (+ 8349us) negotiation.cc:107] Waiting for socket to connect
0810 23:39:57.816238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:39:57.816478 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:39:57.816996 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:39:57.817001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:39:57.817027 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:39:57.817656 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.817672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.819153 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.819159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.821371 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:39:57.821386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.823147 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.823154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:39:57.823329 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.823366 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:39:57.824162 (+ 796us) client_negotiation.cc:240] Negotiation successful
0810 23:39:57.824297 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8230,"mutex_wait_us":62,"thread_start_us":7983,"threads_started":1}
I20260810 23:39:57.824724 18344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf44dccf8923478c8b4c3a9475344ac1" candidate_uuid: "01e07de2df5e407299e313319f730abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681426e752b423197b2a936921a6785"
I20260810 23:39:57.824927 18344 raft_consensus.cc:3037] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:39:57.825388 18118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf44dccf8923478c8b4c3a9475344ac1" candidate_uuid: "e681426e752b423197b2a936921a6785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e07de2df5e407299e313319f730abd" is_pre_election: true
I20260810 23:39:57.825879 18118 raft_consensus.cc:2379] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e681426e752b423197b2a936921a6785 in current term 1: Already voted for candidate 01e07de2df5e407299e313319f730abd in this term.
I20260810 23:39:57.825840 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf44dccf8923478c8b4c3a9475344ac1" candidate_uuid: "e681426e752b423197b2a936921a6785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909dcebcc146402282b1fd30a0a31183" is_pre_election: true
I20260810 23:39:57.826989 18238 leader_election.cc:302] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e681426e752b423197b2a936921a6785; no voters: 01e07de2df5e407299e313319f730abd, 909dcebcc146402282b1fd30a0a31183
I20260810 23:39:57.828353 18344 raft_consensus.cc:2454] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e07de2df5e407299e313319f730abd in term 1.
I20260810 23:39:57.828506 18589 raft_consensus.cc:2454] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e07de2df5e407299e313319f730abd in term 1.
I20260810 23:39:57.828564 18731 raft_consensus.cc:2726] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:39:57.828595 18205 tablet.cc:2369] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:39:57.828899 18012 leader_election.cc:302] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e07de2df5e407299e313319f730abd, 909dcebcc146402282b1fd30a0a31183; no voters:
I20260810 23:39:57.829181 18730 raft_consensus.cc:2781] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:39:57.829636 18730 raft_consensus.cc:686] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 1 LEADER]: Becoming Leader. State: Replica: 01e07de2df5e407299e313319f730abd, State: Running, Role: LEADER
I20260810 23:39:57.830092 18730 consensus_queue.cc:237] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } }
I20260810 23:39:57.835136 17852 catalog_manager.cc:5495] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e07de2df5e407299e313319f730abd (127.30.33.1). New cstate: current_term: 1 leader_uuid: "01e07de2df5e407299e313319f730abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01e07de2df5e407299e313319f730abd" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 } health_report { overall_health: UNKNOWN } } }
I20260810 23:39:57.858403 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.852813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.30.33.1:42639)
0810 23:39:57.853007 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.853013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.853035 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:39:57.853230 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.853234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.853360 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.853502 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:39:57.853903 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.854684 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.856123 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.856554 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.856668 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.857185 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:39:57.857744 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:39:57.857885 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.857990 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.858082 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:39:57.866808 18589 raft_consensus.cc:1261] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 1 FOLLOWER]: Refusing update from remote peer 01e07de2df5e407299e313319f730abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:57.866859 18344 raft_consensus.cc:1261] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 1 FOLLOWER]: Refusing update from remote peer 01e07de2df5e407299e313319f730abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:39:57.867782 18738 consensus_queue.cc:1035] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e681426e752b423197b2a936921a6785" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:57.868135 18730 consensus_queue.cc:1035] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "909dcebcc146402282b1fd30a0a31183" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:39:57.881594 18752 mvcc.cc:205] Tried to move back new op lower bound from 7317115690452574208 to 7317115690315788288. Current Snapshot: MvccSnapshot[applied={T|T < 7317115690452574208}]
I20260810 23:39:57.904771 17850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:57.905122 17850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:39:57.909834 17850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 168290d7c42c4d3d97c2f9fbef0269c7: Sending DeleteTablet for 3 replicas of tablet bf44dccf8923478c8b4c3a9475344ac1
I20260810 23:39:57.911350 18552 tablet_service.cc:1514] Processing DeleteTablet for tablet bf44dccf8923478c8b4c3a9475344ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:57 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:36906
I20260810 23:39:57.911602 18084 tablet_service.cc:1514] Processing DeleteTablet for tablet bf44dccf8923478c8b4c3a9475344ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:57 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:55478
I20260810 23:39:57.912163 18310 tablet_service.cc:1514] Processing DeleteTablet for tablet bf44dccf8923478c8b4c3a9475344ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:39:57 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:43000
I20260810 23:39:57.914064 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 17968
I20260810 23:39:57.917569 18771 tablet_replica.cc:330] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: stopping tablet replica
I20260810 23:39:57.918098 18771 raft_consensus.cc:2229] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:39:57.918779 18771 raft_consensus.cc:2258] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:57.919555 18772 tablet_replica.cc:330] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: stopping tablet replica
I20260810 23:39:57.920030 18772 raft_consensus.cc:2229] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:57.920581 18772 raft_consensus.cc:2258] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:57.922942 18772 ts_tablet_manager.cc:1905] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:57.924973 18771 ts_tablet_manager.cc:1905] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:39:57.930348 18772 ts_tablet_manager.cc:1918] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:57.930567 18772 log.cc:1196] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/bf44dccf8923478c8b4c3a9475344ac1
I20260810 23:39:57.931159 18772 ts_tablet_manager.cc:1939] T bf44dccf8923478c8b4c3a9475344ac1 P e681426e752b423197b2a936921a6785: Deleting consensus metadata
I20260810 23:39:57.933481 17826 catalog_manager.cc:4841] TS e681426e752b423197b2a936921a6785 (127.30.33.2:41151): tablet bf44dccf8923478c8b4c3a9475344ac1 (table test-table [id=c3fd1bfc15894cfe898088614e90edf0]) successfully deleted
I20260810 23:39:57.938486 18771 ts_tablet_manager.cc:1918] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:57.938841 18771 log.cc:1196] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/bf44dccf8923478c8b4c3a9475344ac1
I20260810 23:39:57.939584 18771 ts_tablet_manager.cc:1939] T bf44dccf8923478c8b4c3a9475344ac1 P 01e07de2df5e407299e313319f730abd: Deleting consensus metadata
I20260810 23:39:57.940703 18768 tablet_replica.cc:330] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: stopping tablet replica
I20260810 23:39:57.941226 18768 raft_consensus.cc:2229] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:39:57.941784 18768 raft_consensus.cc:2258] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:39:57.941928 17830 catalog_manager.cc:4841] TS 01e07de2df5e407299e313319f730abd (127.30.33.1:42639): tablet bf44dccf8923478c8b4c3a9475344ac1 (table test-table [id=c3fd1bfc15894cfe898088614e90edf0]) successfully deleted
I20260810 23:39:57.944550 18768 ts_tablet_manager.cc:1905] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:39:57 dist-test-slave-pww0 krb5kdc[17770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:39:57.951156 18768 ts_tablet_manager.cc:1918] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:39:57.951368 18768 log.cc:1196] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/bf44dccf8923478c8b4c3a9475344ac1
I20260810 23:39:57.951987 18768 ts_tablet_manager.cc:1939] T bf44dccf8923478c8b4c3a9475344ac1 P 909dcebcc146402282b1fd30a0a31183: Deleting consensus metadata
I20260810 23:39:57.954142 17828 catalog_manager.cc:4841] TS 909dcebcc146402282b1fd30a0a31183 (127.30.33.3:40027): tablet bf44dccf8923478c8b4c3a9475344ac1 (table test-table [id=c3fd1bfc15894cfe898088614e90edf0]) successfully deleted
I20260810 23:39:57.956897 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:57.915270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.30.33.1:42639)
0810 23:39:57.915480 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:57.915486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:57.934253 (+ 18767us) server_negotiation.cc:407] Connection header received
0810 23:39:57.934766 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:57.934771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:57.934884 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:57.935010 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:57.937318 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.938429 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.942196 (+ 3767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:57.942795 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:57.942920 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:57.948782 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:57.948802 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:57.948822 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:57.948878 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:57.953361 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:57.954543 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:57.954551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:57.954554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:57.954715 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:57.955158 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:57.955161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:57.955163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:57.955661 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:57.955921 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:57.956390 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 23:39:57.956619 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:39:57.957737 18194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:58.062392 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 17968
I20260810 23:39:58.087294 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 18212
Aug 10 23:39:58 dist-test-slave-pww0 krb5kdc[17770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:39:58.103531 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:58.088087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.30.33.2:41151)
0810 23:39:58.088260 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:58.088264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:58.088284 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:39:58.088508 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:58.088513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:58.088629 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:58.088727 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:58.089971 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:58.090746 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:58.092072 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:58.092500 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:58.092602 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:58.097070 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:58.097089 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:58.097107 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:58.097150 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:58.100410 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:58.101234 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:58.101243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:58.101245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:58.101407 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:58.101852 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:58.101856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:58.101858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:58.102273 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:58.102577 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:58.102904 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 23:39:58.103134 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:39:58.104230 18417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:58.198258 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 18212
I20260810 23:39:58.224171 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 18433
Aug 10 23:39:58 dist-test-slave-pww0 krb5kdc[17770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:39:58.239826 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:39:58.224984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.30.33.3:40027)
0810 23:39:58.225177 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:39:58.225182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:39:58.225203 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:39:58.225379 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:39:58.225383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:39:58.225465 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:39:58.225551 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:39:58.226851 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:58.227624 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:58.229057 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:39:58.229542 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:39:58.229667 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:39:58.234213 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:39:58.234234 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:39:58.234258 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:39:58.234301 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:39:58.237063 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:58.237796 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:58.237803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:58.237806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:58.237943 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:39:58.238278 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:39:58.238280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:39:58.238282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:39:58.238659 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:39:58.238893 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:39:58.239264 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:39:58.239505 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:39:58.240489 18663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:58.331323 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 18433
I20260810 23:39:58.355161 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 17801
I20260810 23:39:58.356616 17920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:39:58.469141 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 17801
2026-08-10T23:39:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4925 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:39:58 dist-test-slave-pww0 krb5kdc[18785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:39:58 dist-test-slave-pww0 krb5kdc[18785](info): set up 2 sockets
Aug 10 23:39:58 dist-test-slave-pww0 krb5kdc[18785](info): commencing operation
krb5kdc: starting...
W20260810 23:39:59.798389 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:40:00 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:40:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/rangeradmin_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/rangeradmin_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/rangerlookup_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/rangerlookup_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
I20260810 23:40:00.255055 30852 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdkhI7B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/postgres -l logfile start
2026-08-10 23:40:02.896 UTC [18842] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:40:02.896 UTC [18842] LOG: listening on IPv4 address "127.30.33.20", port 39381
2026-08-10 23:40:02.899 UTC [18842] LOG: listening on Unix socket "/tmp/.s.PGSQL.39381"
2026-08-10 23:40:02.921 UTC [18846] LOG: database system was shut down at 2026-08-10 23:40:01 UTC
2026-08-10 23:40:02.925 UTC [18842] LOG: database system is ready to accept connections
I20260810 23:40:04.083803 30852 mini_postgres.cc:96] Postgres bound to 39381
2026-08-10 23:40:04.110 UTC [18863] FATAL: database "slave" does not exist
127.30.33.20:39381 - accepting connections
I20260810 23:40:04.111177 30852 mini_ranger.cc:161] Starting Ranger...
I20260810 23:40:04.147262 30852 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:40:04.634761 30852 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:40:04.634882 30852 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-admin
2026-08-10 23:40:05,044 [I] DB FLAVOR :POSTGRES
2026-08-10 23:40:05,045 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:40:05,045 [I] Checking connection..
2026-08-10 23:40:05,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:40:05,342 [I] Checking connection passed.
2026-08-10 23:40:05,342 [I] --------- Verifying version history table ---------
2026-08-10 23:40:05,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:40:05,608 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:40:05,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:40:05,855 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:40:05,855 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:40:05,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:40:06.127 UTC [18990] WARNING: there is no transaction in progress
2026-08-10 23:40:06,136 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:40:06,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:40:06,413 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:40:06,413 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:40:06,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:40:06,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:40:06,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pww0.c.gcp-upstream.internal','N') ;"
2026-08-10 23:40:07,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:40:07,451 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:40:07,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:40:07,710 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:40:07,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:40:07,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:40:08,222 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:40:08,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:40:09.243 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.260 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.276 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.292 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.311 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.333 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.400 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.408 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.423 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.434 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.444 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.459 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.469 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.475 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.482 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:09.489 UTC [19182] WARNING: there is no transaction in progress
2026-08-10 23:40:10,152 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:40:10,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:40:10,410 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:40:10,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:40:10,687 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:40:10,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:40:10,944 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:40:10,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:40:11,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:40:11,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pww0.c.gcp-upstream.internal';"
2026-08-10 23:40:11,714 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:40:11,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:40:11,974 [I] DB_PATCHES have already been applied
I20260810 23:40:11.980778 30852 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdkhI7B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:40:11.980873 30852 mini_ranger.cc:193] Using host: 127.30.33.20
I20260810 23:40:12.005848 30852 mini_ranger.cc:227] Ranger admin URL: http://127.30.33.20:48411
Aug 10, 2026 11:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:40:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:40:13.405440 30852 mini_ranger.cc:160] Time spent starting Ranger: real 9.294s user 0.002s sys 0.079s
Aug 10, 2026 11:40:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:40:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 11:40:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:40:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:40:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:40:37.300575 30852 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/rangerkms_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/rangerkms_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.33.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.33.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:40:37 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405237, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:40:37.542493 30852 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:40:37.573519 30852 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:40:38.439103 30852 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:40:38.439276 30852 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms
I20260810 23:40:38.439410 30852 mini_ranger_kms.cc:227] Using postgres at 127.30.33.20:39381
2026-08-10 23:40:39,149 [I] DB FLAVOR :POSTGRES
2026-08-10 23:40:39,149 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:40:39,149 [I] Checking connection
2026-08-10 23:40:39,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:40:39,466 [I] connection success
2026-08-10 23:40:39,466 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:40:39,466 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:40:39,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:40:39,789 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:40:39,790 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:40:39,790 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:40:39,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:39381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:40:40,066 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:40:40.564706 30852 mini_ranger_kms.cc:310] Created kms service
I20260810 23:40:40.750829 30852 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:40:40.796514 30852 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:40:40 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405237, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:41.228536 30852 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:40:41.229820 30852 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:40:41.229897 30852 mini_ranger_kms.cc:240] Using host: 127.30.33.20
I20260810 23:40:41.264371 30852 mini_ranger_kms.cc:276] Ranger KMS PID: 19534
I20260810 23:40:41.264555 30852 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.33.20:34905
Aug 10, 2026 11:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:40:43 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786405243, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.33.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 23:40:44 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405243, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.33.20@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:44.037919 30852 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.495s user 0.176s sys 0.453s
I20260810 23:40:44.038326 30852 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:40:44.038553 30852 mini_ranger_kms.cc:401] 127.30.33.20:34905/kms/v1/keys
Aug 10 23:40:44 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405243, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.33.20@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
Aug 10 23:40:46 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:40:46.901520 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37231
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:37231
--ranger_config_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:34905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:40:47.301661 19618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:40:47.302026 19618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:40:47.302172 19618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:40:47.309808 19618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:40:47.309964 19618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:40:47.310047 19618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:40:47.310129 19618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:40:47.310204 19618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:40:47.321089 19618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.33.20:34905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:37231
--ranger_config_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:40:47.322542 19618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:40:47.324013 19618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:40:47.337265 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:47.337292 19637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:47.337324 19634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:40:47.337739 19618 server_base.cc:1034] running on GCE node
I20260810 23:40:47.338857 19618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:40:47.340510 19618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:40:47.341734 19618 hybrid_clock.cc:648] HybridClock initialized: now 1786405247341679 us; error 33 us; skew 500 ppm
I20260810 23:40:47.342119 19618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:40:47 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405247, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:40:47.349311 19618 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:40:47.351217 19618 webserver.cc:460] Webserver started at http://127.30.33.62:39779/ using document root <none> and password file <none>
I20260810 23:40:47.352032 19618 fs_manager.cc:359] Metadata directory not provided
I20260810 23:40:47.352167 19618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:40:47.352504 19618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:40:47 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405247, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:47.424868 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "bf8843c3f1834405924a96c6d06d45ac"
format_stamp: "Formatted at 2026-08-10 23:40:47 on dist-test-slave-pww0"
server_key: "f8fe265ec8300e2f5b8a8d66b4b187ac"
server_key_iv: "40d9d1f64f01ff3359664117543b9e40"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:47.426047 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "bf8843c3f1834405924a96c6d06d45ac"
format_stamp: "Formatted at 2026-08-10 23:40:47 on dist-test-slave-pww0"
server_key: "f8fe265ec8300e2f5b8a8d66b4b187ac"
server_key_iv: "40d9d1f64f01ff3359664117543b9e40"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:47.433768 19618 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260810 23:40:47.453958 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:47.455605 19618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:40:47.455811 19618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "bf8843c3f1834405924a96c6d06d45ac"
format_stamp: "Formatted at 2026-08-10 23:40:47 on dist-test-slave-pww0"
server_key: "f8fe265ec8300e2f5b8a8d66b4b187ac"
server_key_iv: "40d9d1f64f01ff3359664117543b9e40"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:47.456033 19618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:40:47.473176 19618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:40:47.477581 19618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:40:47.477888 19618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:40:47.496651 19618 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:37231
I20260810 23:40:47.496698 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:37231 every 8 connection(s)
I20260810 23:40:47.498119 19618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:40:47.502413 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:40:47.506860 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 19618
I20260810 23:40:47.507079 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:40:47.507797 30852 external_mini_cluster.cc:1428] Setting key d2d40c74e21a240571a0a74c9e9bad86
I20260810 23:40:47.514219 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac: Bootstrap starting.
I20260810 23:40:47.518133 19744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac: Neither blocks nor log segments found. Creating new log.
I20260810 23:40:47.519397 19744 log.cc:824] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:40:47 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:40:47.522760 19744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac: No bootstrap required, opened a new log
I20260810 23:40:47.531237 19744 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8843c3f1834405924a96c6d06d45ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37231 } }
I20260810 23:40:47.531751 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:47.509893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.30.33.62:37231)
0810 23:40:47.510818 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:47.510835 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:47.510882 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:40:47.512731 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:47.512757 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:47.513193 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:47.513548 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:47.515271 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:47.516749 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:47.518511 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:47.519249 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:47.519402 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:47.523651 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:47.523660 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:47.523676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:47.523722 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:47.526838 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:47.527678 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:47.527686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:47.527694 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:47.527893 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:47.528324 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:47.528329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:47.528331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:47.528975 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:47.529334 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:47.529839 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 23:40:47.530353 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":436,"threads_started":1}
I20260810 23:40:47.531867 19744 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:40:47.532016 19744 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8843c3f1834405924a96c6d06d45ac, State: Initialized, Role: FOLLOWER
I20260810 23:40:47.532748 19744 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8843c3f1834405924a96c6d06d45ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37231 } }
I20260810 23:40:47.533007 19744 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:40:47.533201 19744 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:40:47.533390 19744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:40:47.536093 19744 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8843c3f1834405924a96c6d06d45ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37231 } }
I20260810 23:40:47.536737 19744 leader_election.cc:302] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf8843c3f1834405924a96c6d06d45ac; no voters:
I20260810 23:40:47.537395 19744 leader_election.cc:288] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:40:47.537483 19752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:40:47.538163 19752 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [term 1 LEADER]: Becoming Leader. State: Replica: bf8843c3f1834405924a96c6d06d45ac, State: Running, Role: LEADER
I20260810 23:40:47.538720 19752 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8843c3f1834405924a96c6d06d45ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37231 } }
I20260810 23:40:47.539541 19744 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:40:47.541963 19753 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf8843c3f1834405924a96c6d06d45ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8843c3f1834405924a96c6d06d45ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37231 } } }
I20260810 23:40:47.542433 19753 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:40:47.542011 19754 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf8843c3f1834405924a96c6d06d45ac. Latest consensus state: current_term: 1 leader_uuid: "bf8843c3f1834405924a96c6d06d45ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8843c3f1834405924a96c6d06d45ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37231 } } }
I20260810 23:40:47.543004 19754 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:40:47.544139 19768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:40:47.549677 19768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:40:47.551720 19744 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:40:47.569301 19768 catalog_manager.cc:1347] Generated new cluster ID: 7da95ed6c1c44b088507903cf1ce4a3a
I20260810 23:40:47.569479 19768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:40:47.591421 19768 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:40:47.592993 19768 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:40:47.609632 19768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac: Generated new TSK 0
I20260810 23:40:47.611287 19768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:40:48.419375 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:34905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37231
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:40:48 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786405248, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:40:48.944929 19821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:40:48.945441 19821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:40:48.945645 19821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:40:48.955415 19821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:40:48.955657 19821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:40:48.955847 19821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:40:48.969071 19821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.33.20:34905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:40:48.970570 19821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:40:48.972121 19821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:40:48.995164 19844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:48.998867 19841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:49.000826 19846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:40:49.001446 19821 server_base.cc:1034] running on GCE node
I20260810 23:40:49.002204 19821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:40:49.003285 19821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:40:49.004511 19821 hybrid_clock.cc:648] HybridClock initialized: now 1786405249004436 us; error 64 us; skew 500 ppm
I20260810 23:40:49.004879 19821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:40:49 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405249, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:40:49.008318 19821 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:40:49.010115 19821 webserver.cc:460] Webserver started at http://127.30.33.1:34849/ using document root <none> and password file <none>
I20260810 23:40:49.010917 19821 fs_manager.cc:359] Metadata directory not provided
I20260810 23:40:49.011032 19821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:40:49.011363 19821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:40:49 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405249, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:49.030946 19821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "e114d6d3f8a74af4ac42a5c5991f7c39"
format_stamp: "Formatted at 2026-08-10 23:40:49 on dist-test-slave-pww0"
server_key: "dd7bbbb2532ecbba24c912ec33c0520b"
server_key_iv: "5899214ece7859b233de3cb0b8996379"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:49.032135 19821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "e114d6d3f8a74af4ac42a5c5991f7c39"
format_stamp: "Formatted at 2026-08-10 23:40:49 on dist-test-slave-pww0"
server_key: "dd7bbbb2532ecbba24c912ec33c0520b"
server_key_iv: "5899214ece7859b233de3cb0b8996379"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:49.044623 19821 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.010s sys 0.001s
I20260810 23:40:49.074891 19857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:49.077306 19821 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260810 23:40:49.077556 19821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "e114d6d3f8a74af4ac42a5c5991f7c39"
format_stamp: "Formatted at 2026-08-10 23:40:49 on dist-test-slave-pww0"
server_key: "dd7bbbb2532ecbba24c912ec33c0520b"
server_key_iv: "5899214ece7859b233de3cb0b8996379"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:49.077860 19821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:40:49.124826 19821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:40:49.134176 19821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:40:49.134665 19821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:40:49.145762 19821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:40:49.154686 19821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:40:49.154893 19821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 23:40:49.155231 19821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:40:49.155438 19821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:40:49 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405248, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:49.728127 19821 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:37467
I20260810 23:40:49.728161 20075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:37467 every 8 connection(s)
I20260810 23:40:49.730017 19821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:40:49 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405248, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:49.735069 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 19821
I20260810 23:40:49.735297 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:40:49.736189 30852 external_mini_cluster.cc:1428] Setting key f75191987904e1900ee338c619ea7821
Aug 10 23:40:49 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405249, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:40:49.774470 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:49.736676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37231 (local address 127.30.33.1:51781)
0810 23:40:49.747877 (+ 11201us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:49.747940 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:49.750720 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:49.752238 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:49.752253 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:49.752983 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:40:49.753927 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:49.753967 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:49.755684 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:49.755690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:49.757850 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:49.757870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:49.759255 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:49.759266 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:49.759634 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:49.760624 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:40:49.760658 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:40:49.763793 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:40:49.768509 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:40:49.768523 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:40:49.768546 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:40:49.769300 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:40:49.769887 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:40:49.769893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:40:49.769896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:40:49.770281 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:40:49.771234 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:40:49.771244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:40:49.771813 (+ 569us) client_negotiation.cc:769] Sending connection context
0810 23:40:49.772185 (+ 372us) client_negotiation.cc:240] Negotiation successful
0810 23:40:49.772856 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10833,"thread_start_us":10537,"threads_started":1}
I20260810 23:40:49.776058 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:49.736135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:51781 (local address 127.30.33.62:37231)
0810 23:40:49.751381 (+ 15246us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:49.751388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:49.751409 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:40:49.751496 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:49.751502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:49.751598 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:49.751710 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:49.754204 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:49.755424 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:49.758055 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:49.758668 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:49.758847 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:49.764174 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:49.764183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:49.764188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:49.764241 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:49.768232 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:49.769498 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:49.769505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:49.769508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:49.769692 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:49.770447 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:49.770452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:49.770455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:49.770938 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:49.771125 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:49.775317 (+ 4192us) server_negotiation.cc:299] Negotiation successful
0810 23:40:49.775654 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15077,"thread_start_us":19480,"threads_started":1}
I20260810 23:40:49.777886 20076 heartbeater.cc:347] Connected to a master server at 127.30.33.62:37231
I20260810 23:40:49.778296 20076 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
I20260810 23:40:49.779314 20076 heartbeater.cc:511] Master 127.30.33.62:37231 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
I20260810 23:40:49.782397 19677 ts_manager.cc:194] Registered new tserver with Master: e114d6d3f8a74af4ac42a5c5991f7c39 (127.30.33.1:37467)
I20260810 23:40:49.785019 19677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:51781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:40:49.822791 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:34905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37231
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:40:50.255208 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:40:50.255570 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:40:50.255723 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:40:50.263518 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:40:50.263644 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:40:50.263796 20087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:40:50.274165 20087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.33.20:34905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:40:50.275622 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:40:50.277025 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:40:50.292809 20109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:50.292979 20108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:50.292835 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:40:50.294056 20087 server_base.cc:1034] running on GCE node
I20260810 23:40:50.294750 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:40:50.295926 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:40:50.297158 20087 hybrid_clock.cc:648] HybridClock initialized: now 1786405250297089 us; error 66 us; skew 500 ppm
I20260810 23:40:50.297509 20087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:40:50 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405250, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:40:50.300912 20087 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:40:50.302621 20087 webserver.cc:460] Webserver started at http://127.30.33.2:42567/ using document root <none> and password file <none>
I20260810 23:40:50.303363 20087 fs_manager.cc:359] Metadata directory not provided
I20260810 23:40:50.303530 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:40:50.303869 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:40:50 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405250, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:50.322528 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "b78dad473091426c9d275337bf3ca07e"
format_stamp: "Formatted at 2026-08-10 23:40:50 on dist-test-slave-pww0"
server_key: "fcd1a7c809b9d8c1fd1d2d55c0504860"
server_key_iv: "31e2c94a1e5156e3d0062fca7683eb17"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:50.323660 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "b78dad473091426c9d275337bf3ca07e"
format_stamp: "Formatted at 2026-08-10 23:40:50 on dist-test-slave-pww0"
server_key: "fcd1a7c809b9d8c1fd1d2d55c0504860"
server_key_iv: "31e2c94a1e5156e3d0062fca7683eb17"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:50.329849 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:40:50.348920 20126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:50.350728 20087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:40:50.350940 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "b78dad473091426c9d275337bf3ca07e"
format_stamp: "Formatted at 2026-08-10 23:40:50 on dist-test-slave-pww0"
server_key: "fcd1a7c809b9d8c1fd1d2d55c0504860"
server_key_iv: "31e2c94a1e5156e3d0062fca7683eb17"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:50.351176 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:40:50.386025 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:40:50.390841 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:40:50.391168 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:40:50.392347 20087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:40:50.394081 20087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:40:50.394172 20087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:50.394302 20087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:40:50.394362 20087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:50.427949 20087 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:41661
I20260810 23:40:50.427965 20313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:41661 every 8 connection(s)
I20260810 23:40:50.429303 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:40:50.434898 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 20087
I20260810 23:40:50.435065 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:40:50.435527 30852 external_mini_cluster.cc:1428] Setting key d6fb8de22393f2ebd737077fea7a624a
Aug 10 23:40:50 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405250, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:40:50.451342 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:50.431989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:33421 (local address 127.30.33.62:37231)
0810 23:40:50.432486 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:50.432491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:50.433616 (+ 1125us) server_negotiation.cc:407] Connection header received
0810 23:40:50.435544 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:50.435549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:50.435642 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:50.435751 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:50.437456 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:50.438555 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:50.440337 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:50.440862 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:50.440990 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:50.444602 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:50.444610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:50.444615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:50.444661 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:50.448021 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:50.448829 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:50.448835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:50.448839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:50.448995 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:50.449438 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:50.449442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:50.449444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:50.449801 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:50.449965 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:50.450865 (+ 900us) server_negotiation.cc:299] Negotiation successful
0810 23:40:50.451057 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
I20260810 23:40:50.452813 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:50.432646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37231 (local address 127.30.33.2:33421)
0810 23:40:50.433438 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:50.433479 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:50.435237 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:50.436023 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:50.436033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:50.436587 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:40:50.437266 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:50.437308 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:50.438736 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:50.438740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:50.440157 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:50.440172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:50.441066 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:50.441071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:50.441295 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:50.441963 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:40:50.441988 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:40:50.444329 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:40:50.448187 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:40:50.448196 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:40:50.448214 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:40:50.448686 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:40:50.449105 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:40:50.449108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:40:50.449111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:40:50.449339 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:40:50.449951 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:40:50.449960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:40:50.450436 (+ 476us) client_negotiation.cc:769] Sending connection context
0810 23:40:50.450773 (+ 337us) client_negotiation.cc:240] Negotiation successful
0810 23:40:50.451362 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":324,"threads_started":1}
I20260810 23:40:50.454701 20314 heartbeater.cc:347] Connected to a master server at 127.30.33.62:37231
I20260810 23:40:50.455127 20314 heartbeater.cc:464] Registering TS with master...
I20260810 23:40:50.456108 20314 heartbeater.cc:511] Master 127.30.33.62:37231 requested a full tablet report, sending...
I20260810 23:40:50.457880 19677 ts_manager.cc:194] Registered new tserver with Master: b78dad473091426c9d275337bf3ca07e (127.30.33.2:41661)
I20260810 23:40:50.458966 19677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:33421
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:40:50.502866 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:34905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37231
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:40:50.789413 20076 heartbeater.cc:503] Master 127.30.33.62:37231 was elected leader, sending a full tablet report...
W20260810 23:40:50.860677 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:40:50.860999 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:40:50.861128 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:40:50.868510 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:40:50.868634 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:40:50.868768 20321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:40:50.879040 20321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.33.20:34905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:40:50.880460 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:40:50.881835 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:40:50.896497 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:50.896426 20349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:40:50.896463 20339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:40:50.897289 20321 server_base.cc:1034] running on GCE node
I20260810 23:40:50.897985 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:40:50.899070 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:40:50.900302 20321 hybrid_clock.cc:648] HybridClock initialized: now 1786405250900238 us; error 54 us; skew 500 ppm
I20260810 23:40:50.900676 20321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:40:50 dist-test-slave-pww0 krb5kdc[18785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405250, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:40:50.904022 20321 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:40:50.905601 20321 webserver.cc:460] Webserver started at http://127.30.33.3:40981/ using document root <none> and password file <none>
I20260810 23:40:50.906358 20321 fs_manager.cc:359] Metadata directory not provided
I20260810 23:40:50.906481 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:40:50.906828 20321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:40:50 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405250, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:40:50.922279 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "78934a3b2220497c8338eb6a1c524b98"
format_stamp: "Formatted at 2026-08-10 23:40:50 on dist-test-slave-pww0"
server_key: "3482718ab466ebb883c0270a0a65351c"
server_key_iv: "1e61c09dfc0464d9ff58ee162ab17b3d"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:50.923202 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "78934a3b2220497c8338eb6a1c524b98"
format_stamp: "Formatted at 2026-08-10 23:40:50 on dist-test-slave-pww0"
server_key: "3482718ab466ebb883c0270a0a65351c"
server_key_iv: "1e61c09dfc0464d9ff58ee162ab17b3d"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:50.929571 20321 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 23:40:50.948683 20360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:50.950383 20321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:40:50.950588 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "78934a3b2220497c8338eb6a1c524b98"
format_stamp: "Formatted at 2026-08-10 23:40:50 on dist-test-slave-pww0"
server_key: "3482718ab466ebb883c0270a0a65351c"
server_key_iv: "1e61c09dfc0464d9ff58ee162ab17b3d"
server_key_version: "kuduclusterkey@0"
I20260810 23:40:50.950819 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:40:50.969381 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:40:50.974215 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:40:50.974548 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:40:50.975700 20321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:40:50.977350 20321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:40:50.977435 20321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:50.977558 20321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:40:50.977609 20321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:40:51.014972 20321 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:42131
I20260810 23:40:51.015008 20584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:42131 every 8 connection(s)
I20260810 23:40:51.016419 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:40:51.023303 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 20321
I20260810 23:40:51.023530 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:40:51.024109 30852 external_mini_cluster.cc:1428] Setting key 1ea85ba09e4cc192a9ea0d20204f1f36
Aug 10 23:40:51 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405250, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:40:51.039274 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.019494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:53573 (local address 127.30.33.62:37231)
0810 23:40:51.020076 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.020082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.021122 (+ 1040us) server_negotiation.cc:407] Connection header received
0810 23:40:51.023064 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.023070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.023169 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.023279 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:51.025470 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.026252 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.028027 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.028472 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.028590 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.032290 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:51.032296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:51.032300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:51.032337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:51.035399 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.036396 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.036402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.036404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.036557 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.037101 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.037106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.037109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.037427 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:51.037591 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.038523 (+ 932us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.038892 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":300,"threads_started":1}
I20260810 23:40:51.040572 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.020045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37231 (local address 127.30.33.3:53573)
0810 23:40:51.020912 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:51.020955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:51.022692 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:51.023675 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:51.023688 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:51.024355 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:40:51.025263 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.025296 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.026434 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.026438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.027869 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.027885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.028749 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.028756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.029013 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.029711 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:40:51.029736 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:40:51.032007 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:40:51.035635 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:40:51.035646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:40:51.035664 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:40:51.036233 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:40:51.036694 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:40:51.036699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:40:51.036703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:40:51.036960 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:40:51.037589 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:40:51.037599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:40:51.037984 (+ 385us) client_negotiation.cc:769] Sending connection context
0810 23:40:51.038386 (+ 402us) client_negotiation.cc:240] Negotiation successful
0810 23:40:51.039039 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":365,"threads_started":1}
I20260810 23:40:51.042135 20586 heartbeater.cc:347] Connected to a master server at 127.30.33.62:37231
I20260810 23:40:51.042505 20586 heartbeater.cc:464] Registering TS with master...
I20260810 23:40:51.043316 20586 heartbeater.cc:511] Master 127.30.33.62:37231 requested a full tablet report, sending...
I20260810 23:40:51.045064 19677 ts_manager.cc:194] Registered new tserver with Master: 78934a3b2220497c8338eb6a1c524b98 (127.30.33.3:42131)
I20260810 23:40:51.045923 19677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:53573
I20260810 23:40:51.053264 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:40:51.071693 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.058227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.30.33.62:37231)
0810 23:40:51.058481 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.058489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.058945 (+ 456us) server_negotiation.cc:407] Connection header received
0810 23:40:51.059093 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.059097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.059179 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.059277 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:51.060903 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.061640 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.063149 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.063653 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.063761 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.066256 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:51.066261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:51.066265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:51.066301 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:51.068984 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.069782 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.069787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.069790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.069936 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.070310 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.070313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.070315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.070595 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:51.070790 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.071157 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.071350 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:40:51.076125 19677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:40:51.083501 19677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:40:51.108644 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.097978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.30.33.2:41661)
0810 23:40:51.098627 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.098635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.099234 (+ 599us) server_negotiation.cc:407] Connection header received
0810 23:40:51.099359 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.099366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.099660 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.099872 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:40:51.101560 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.102980 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.105617 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.107046 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.107170 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.107378 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.108105 (+ 727us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.108292 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":1007,"threads_started":1}
I20260810 23:40:51.108692 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.097390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:42131 (local address 127.0.0.1:50064)
0810 23:40:51.098718 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:51.098762 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:51.099066 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:51.100167 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:51.100175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:51.100239 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:40:51.100820 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.100835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.102831 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.102836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.105153 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.105167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.107690 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.107698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.107885 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.107911 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:40:51.108012 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:40:51.108264 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"spinlock_wait_cycles":3840,"thread_start_us":892,"threads_started":1}
I20260810 23:40:51.109035 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.097496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.30.33.3:42131)
0810 23:40:51.098260 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.098270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.098884 (+ 614us) server_negotiation.cc:407] Connection header received
0810 23:40:51.099335 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.099343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.099649 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.099826 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:40:51.101082 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.102698 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.105329 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.106837 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.106979 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.107259 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.108477 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.108666 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":437,"threads_started":1}
I20260810 23:40:51.109030 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.099432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37467 (local address 127.0.0.1:41806)
0810 23:40:51.100726 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:51.100743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:51.100863 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:51.102029 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:51.102034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:51.102058 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:40:51.102438 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.102448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.104473 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.104479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.106269 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.106281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.108273 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.108279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.108458 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.108491 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:40:51.108575 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:40:51.108721 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":994,"threads_started":1}
I20260810 23:40:51.109652 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.097508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41661 (local address 127.0.0.1:60162)
0810 23:40:51.099067 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:51.099084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:51.099192 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:51.100167 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:51.100176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:51.100234 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:40:51.101366 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.101380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.103143 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.103147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.105425 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.105441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.107445 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.107451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.107801 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.107872 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 23:40:51.109204 (+ 1332us) client_negotiation.cc:240] Negotiation successful
0810 23:40:51.109357 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"mutex_wait_us":58,"spinlock_wait_cycles":1408,"thread_start_us":1303,"threads_started":1}
I20260810 23:40:51.110020 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.100117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.30.33.1:37467)
0810 23:40:51.101525 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.101533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.101558 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:40:51.101645 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.101650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.101849 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.102022 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:40:51.102613 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.104286 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.106429 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.107740 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.107866 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.108107 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.109344 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.109528 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":1071,"threads_started":1}
I20260810 23:40:51.113282 20463 tablet_service.cc:1467] Processing CreateTablet for tablet 528c1f70d8d042e49e5bcf6167b70276 (DEFAULT_TABLE table=test-table [id=435ba32f096642eeb11b9d1ad5b0a12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:40:51.113369 19943 tablet_service.cc:1467] Processing CreateTablet for tablet 528c1f70d8d042e49e5bcf6167b70276 (DEFAULT_TABLE table=test-table [id=435ba32f096642eeb11b9d1ad5b0a12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:40:51.113314 20218 tablet_service.cc:1467] Processing CreateTablet for tablet 528c1f70d8d042e49e5bcf6167b70276 (DEFAULT_TABLE table=test-table [id=435ba32f096642eeb11b9d1ad5b0a12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:40:51.114594 20463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 528c1f70d8d042e49e5bcf6167b70276. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:40:51.114739 20218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 528c1f70d8d042e49e5bcf6167b70276. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:40:51.114770 19943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 528c1f70d8d042e49e5bcf6167b70276. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:40:51.138317 20621 tablet_bootstrap.cc:492] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Bootstrap starting.
I20260810 23:40:51.143060 20621 tablet_bootstrap.cc:654] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Neither blocks nor log segments found. Creating new log.
I20260810 23:40:51.144726 20621 log.cc:824] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Log is configured to *not* fsync() on all Append() calls
I20260810 23:40:51.147790 20623 tablet_bootstrap.cc:492] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Bootstrap starting.
I20260810 23:40:51.148053 20622 tablet_bootstrap.cc:492] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Bootstrap starting.
I20260810 23:40:51.150655 20621 tablet_bootstrap.cc:492] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: No bootstrap required, opened a new log
I20260810 23:40:51.150998 20621 ts_tablet_manager.cc:1397] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260810 23:40:51.153251 20622 tablet_bootstrap.cc:654] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Neither blocks nor log segments found. Creating new log.
I20260810 23:40:51.153292 20623 tablet_bootstrap.cc:654] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Neither blocks nor log segments found. Creating new log.
I20260810 23:40:51.154659 20623 log.cc:824] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Log is configured to *not* fsync() on all Append() calls
I20260810 23:40:51.154774 20622 log.cc:824] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:40:51.161866 20621 raft_consensus.cc:348] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.162400 20621 raft_consensus.cc:374] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:40:51.162514 20621 raft_consensus.cc:729] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78934a3b2220497c8338eb6a1c524b98, State: Initialized, Role: FOLLOWER
I20260810 23:40:51.162730 20623 tablet_bootstrap.cc:492] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: No bootstrap required, opened a new log
I20260810 23:40:51.162858 20622 tablet_bootstrap.cc:492] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: No bootstrap required, opened a new log
I20260810 23:40:51.163049 20623 ts_tablet_manager.cc:1397] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260810 23:40:51.163151 20622 ts_tablet_manager.cc:1397] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260810 23:40:51.163368 20621 consensus_queue.cc:260] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.164644 20586 heartbeater.cc:503] Master 127.30.33.62:37231 was elected leader, sending a full tablet report...
I20260810 23:40:51.166852 20621 ts_tablet_manager.cc:1428] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260810 23:40:51.171784 20623 raft_consensus.cc:348] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.172331 20623 raft_consensus.cc:374] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:40:51.172459 20623 raft_consensus.cc:729] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114d6d3f8a74af4ac42a5c5991f7c39, State: Initialized, Role: FOLLOWER
I20260810 23:40:51.173111 20623 consensus_queue.cc:260] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.173514 20622 raft_consensus.cc:348] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.174005 20622 raft_consensus.cc:374] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:40:51.174116 20622 raft_consensus.cc:729] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78dad473091426c9d275337bf3ca07e, State: Initialized, Role: FOLLOWER
I20260810 23:40:51.174778 20622 consensus_queue.cc:260] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.175040 20623 ts_tablet_manager.cc:1428] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260810 23:40:51.176143 20314 heartbeater.cc:503] Master 127.30.33.62:37231 was elected leader, sending a full tablet report...
I20260810 23:40:51.176466 20622 ts_tablet_manager.cc:1428] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
W20260810 23:40:51.181234 20315 tablet.cc:2369] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:40:51.252321 20078 tablet.cc:2369] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:40:51.268103 20587 tablet.cc:2369] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:40:51.275785 20627 raft_consensus.cc:482] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:40:51.276140 20627 raft_consensus.cc:504] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.277990 20627 leader_election.cc:288] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b78dad473091426c9d275337bf3ca07e (127.30.33.2:41661), e114d6d3f8a74af4ac42a5c5991f7c39 (127.30.33.1:37467)
I20260810 23:40:51.284438 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.278457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:52369 (local address 127.30.33.2:41661)
0810 23:40:51.278660 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.278666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.278690 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:40:51.278773 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.278778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.278859 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.278968 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:40:51.279565 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.280522 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.282406 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.283518 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.283631 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.283709 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.284112 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.284227 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:40:51.284511 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.278257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:41661 (local address 127.30.33.3:52369)
0810 23:40:51.278446 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:51.278467 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:51.278597 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:51.279162 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:51.279166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:51.279190 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:40:51.279360 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.279371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.280674 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.280680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.282234 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.282249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.283678 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.283684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.283923 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.283979 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 23:40:51.284080 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:40:51.284223 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:40:51.285342 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.278845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:53031 (local address 127.30.33.1:37467)
0810 23:40:51.279049 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.279054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.279402 (+ 348us) server_negotiation.cc:407] Connection header received
0810 23:40:51.279535 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.279541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.279602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.279669 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:40:51.280286 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.281229 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.283180 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.284324 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.284455 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.284556 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.284973 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.285114 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:40:51.285437 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.278670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37467 (local address 127.30.33.3:53031)
0810 23:40:51.279284 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 23:40:51.279299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:40:51.279405 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:40:51.279704 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:40:51.279726 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:40:51.279753 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:40:51.280131 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.280143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.281397 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.281401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.283031 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:40:51.283046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.284547 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.284553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:40:51.284744 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.284774 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:40:51.284936 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:40:51.285100 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":1,"thread_start_us":317,"threads_started":1}
I20260810 23:40:51.285811 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528c1f70d8d042e49e5bcf6167b70276" candidate_uuid: "78934a3b2220497c8338eb6a1c524b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b78dad473091426c9d275337bf3ca07e" is_pre_election: true
I20260810 23:40:51.286404 20245 raft_consensus.cc:2454] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78934a3b2220497c8338eb6a1c524b98 in term 0.
I20260810 23:40:51.286499 19990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528c1f70d8d042e49e5bcf6167b70276" candidate_uuid: "78934a3b2220497c8338eb6a1c524b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" is_pre_election: true
I20260810 23:40:51.287063 19990 raft_consensus.cc:2454] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78934a3b2220497c8338eb6a1c524b98 in term 0.
I20260810 23:40:51.287181 20365 leader_election.cc:302] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78934a3b2220497c8338eb6a1c524b98, b78dad473091426c9d275337bf3ca07e; no voters:
I20260810 23:40:51.287636 20627 raft_consensus.cc:2781] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:40:51.287776 20627 raft_consensus.cc:482] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:40:51.287895 20627 raft_consensus.cc:3037] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:40:51.290298 20627 raft_consensus.cc:504] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.290920 20627 leader_election.cc:288] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [CANDIDATE]: Term 1 election: Requested vote from peers b78dad473091426c9d275337bf3ca07e (127.30.33.2:41661), e114d6d3f8a74af4ac42a5c5991f7c39 (127.30.33.1:37467)
I20260810 23:40:51.291476 20245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528c1f70d8d042e49e5bcf6167b70276" candidate_uuid: "78934a3b2220497c8338eb6a1c524b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b78dad473091426c9d275337bf3ca07e"
I20260810 23:40:51.291579 19990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528c1f70d8d042e49e5bcf6167b70276" candidate_uuid: "78934a3b2220497c8338eb6a1c524b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39"
I20260810 23:40:51.291739 20245 raft_consensus.cc:3037] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:40:51.291791 19990 raft_consensus.cc:3037] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:40:51.295034 19990 raft_consensus.cc:2454] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78934a3b2220497c8338eb6a1c524b98 in term 1.
I20260810 23:40:51.295034 20245 raft_consensus.cc:2454] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78934a3b2220497c8338eb6a1c524b98 in term 1.
I20260810 23:40:51.295600 20365 leader_election.cc:302] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78934a3b2220497c8338eb6a1c524b98, b78dad473091426c9d275337bf3ca07e; no voters:
I20260810 23:40:51.295912 20627 raft_consensus.cc:2781] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:40:51.296404 20627 raft_consensus.cc:686] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 1 LEADER]: Becoming Leader. State: Replica: 78934a3b2220497c8338eb6a1c524b98, State: Running, Role: LEADER
I20260810 23:40:51.296906 20627 consensus_queue.cc:237] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } }
I20260810 23:40:51.301736 19676 catalog_manager.cc:5495] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78934a3b2220497c8338eb6a1c524b98 (127.30.33.3). New cstate: current_term: 1 leader_uuid: "78934a3b2220497c8338eb6a1c524b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78934a3b2220497c8338eb6a1c524b98" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 42131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 } health_report { overall_health: UNKNOWN } } }
I20260810 23:40:51.358645 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.353148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.30.33.3:42131)
0810 23:40:51.353364 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.353369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.353391 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:40:51.353602 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.353606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.353688 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.353813 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:40:51.354140 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.354945 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.356619 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.357029 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.357126 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.357601 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:40:51.358111 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:40:51.358226 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.358331 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.358400 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:40:51.367255 19990 raft_consensus.cc:1261] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 1 FOLLOWER]: Refusing update from remote peer 78934a3b2220497c8338eb6a1c524b98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:40:51.367244 20245 raft_consensus.cc:1261] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 1 FOLLOWER]: Refusing update from remote peer 78934a3b2220497c8338eb6a1c524b98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:40:51.368264 20636 consensus_queue.cc:1035] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e114d6d3f8a74af4ac42a5c5991f7c39" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:40:51.368690 20627 consensus_queue.cc:1035] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b78dad473091426c9d275337bf3ca07e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 41661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:40:51.382197 20644 mvcc.cc:205] Tried to move back new op lower bound from 7317115909589729280 to 7317115909315866624. Current Snapshot: MvccSnapshot[applied={T|T < 7317115909589729280}]
I20260810 23:40:51.395314 19677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:40:51.395630 19677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:40:51.399350 19677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf8843c3f1834405924a96c6d06d45ac: Sending DeleteTablet for 3 replicas of tablet 528c1f70d8d042e49e5bcf6167b70276
I20260810 23:40:51.400724 20463 tablet_service.cc:1514] Processing DeleteTablet for tablet 528c1f70d8d042e49e5bcf6167b70276 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:40:51 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:50064
I20260810 23:40:51.400818 20218 tablet_service.cc:1514] Processing DeleteTablet for tablet 528c1f70d8d042e49e5bcf6167b70276 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:40:51 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:60162
I20260810 23:40:51.400933 19943 tablet_service.cc:1514] Processing DeleteTablet for tablet 528c1f70d8d042e49e5bcf6167b70276 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:40:51 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:41806
I20260810 23:40:51.403688 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 19821
I20260810 23:40:51.416234 20667 tablet_replica.cc:330] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: stopping tablet replica
I20260810 23:40:51.416896 20667 raft_consensus.cc:2229] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:40:51.417745 20667 raft_consensus.cc:2258] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:40:51.418771 20668 tablet_replica.cc:330] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: stopping tablet replica
I20260810 23:40:51.419330 20668 raft_consensus.cc:2229] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:40:51.419902 20669 tablet_replica.cc:330] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: stopping tablet replica
I20260810 23:40:51.420293 20668 raft_consensus.cc:2258] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:40:51.420351 20669 raft_consensus.cc:2229] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:40:51.420867 20669 raft_consensus.cc:2258] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:40:51.422746 20668 ts_tablet_manager.cc:1905] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:40:51.422772 20667 ts_tablet_manager.cc:1905] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:40:51.430040 20667 ts_tablet_manager.cc:1918] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:40:51.430266 20667 log.cc:1196] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/528c1f70d8d042e49e5bcf6167b70276
I20260810 23:40:51.430394 20668 ts_tablet_manager.cc:1918] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:40:51.430540 20668 log.cc:1196] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/528c1f70d8d042e49e5bcf6167b70276
Aug 10 23:40:51 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:40:51.430841 20667 ts_tablet_manager.cc:1939] T 528c1f70d8d042e49e5bcf6167b70276 P 78934a3b2220497c8338eb6a1c524b98: Deleting consensus metadata
I20260810 23:40:51.431026 20668 ts_tablet_manager.cc:1939] T 528c1f70d8d042e49e5bcf6167b70276 P b78dad473091426c9d275337bf3ca07e: Deleting consensus metadata
I20260810 23:40:51.432955 19654 catalog_manager.cc:4841] TS 78934a3b2220497c8338eb6a1c524b98 (127.30.33.3:42131): tablet 528c1f70d8d042e49e5bcf6167b70276 (table test-table [id=435ba32f096642eeb11b9d1ad5b0a12e]) successfully deleted
I20260810 23:40:51.433109 19650 catalog_manager.cc:4841] TS b78dad473091426c9d275337bf3ca07e (127.30.33.2:41661): tablet 528c1f70d8d042e49e5bcf6167b70276 (table test-table [id=435ba32f096642eeb11b9d1ad5b0a12e]) successfully deleted
I20260810 23:40:51.433642 20669 ts_tablet_manager.cc:1905] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:40:51.440253 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.405301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.30.33.1:37467)
0810 23:40:51.405496 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.405503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.419108 (+ 13605us) server_negotiation.cc:407] Connection header received
0810 23:40:51.419327 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.419333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.419447 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.419609 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:51.421907 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.423040 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.425312 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.425936 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.426040 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.432222 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:51.432237 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:51.432256 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:51.432302 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:51.436228 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.437278 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.437288 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.437291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.437469 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.438015 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.438021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.438024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.438519 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:51.438839 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.439534 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.439830 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:40:51.441133 20669 ts_tablet_manager.cc:1918] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:40:51.441322 20669 log.cc:1196] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/528c1f70d8d042e49e5bcf6167b70276
I20260810 23:40:51.441874 20669 ts_tablet_manager.cc:1939] T 528c1f70d8d042e49e5bcf6167b70276 P e114d6d3f8a74af4ac42a5c5991f7c39: Deleting consensus metadata
I20260810 23:40:51.442386 20063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:40:51.549238 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 19821
I20260810 23:40:51.549440 19650 catalog_manager.cc:4841] TS e114d6d3f8a74af4ac42a5c5991f7c39 (127.30.33.1:37467): tablet 528c1f70d8d042e49e5bcf6167b70276 (table test-table [id=435ba32f096642eeb11b9d1ad5b0a12e]) successfully deleted
I20260810 23:40:51.572443 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 20087
Aug 10 23:40:51 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:40:51.589931 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.573216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.30.33.2:41661)
0810 23:40:51.573406 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.573412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.573448 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:40:51.573648 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.573652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.573740 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.573830 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:51.575287 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.576095 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.577422 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.577806 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.577918 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.582631 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:51.582654 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:51.582675 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:51.582756 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:51.586499 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.587370 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.587382 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.587385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.587812 (+ 427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.588202 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.588207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.588211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.588767 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:51.589062 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.589284 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.589519 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:40:51.590647 20305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:40:51.700606 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 20087
I20260810 23:40:51.725960 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 20321
Aug 10 23:40:51 dist-test-slave-pww0 krb5kdc[18785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:40:51.744853 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:40:51.726779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.30.33.3:42131)
0810 23:40:51.726966 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:40:51.726973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:40:51.727064 (+ 91us) server_negotiation.cc:407] Connection header received
0810 23:40:51.727264 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:40:51.727270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:40:51.727352 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:40:51.727437 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:40:51.728819 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.729638 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.731010 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:40:51.731543 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:40:51.731666 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:40:51.736681 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:40:51.736707 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:40:51.736741 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:40:51.736812 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:40:51.741125 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.742115 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.742125 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.742128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.742342 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:40:51.742847 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:40:51.742853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:40:51.742855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:40:51.743328 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:40:51.743717 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0810 23:40:51.744157 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 23:40:51.744415 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:40:51.745648 20576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:40:51.872159 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 20321
I20260810 23:40:51.893643 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 19618
I20260810 23:40:51.894932 19733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:40:52.009298 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 19618
I20260810 23:40:52.039855 30852 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:40:52.387300 30852 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:40:52.387656 30852 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:40:52.744478 30852 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:40:52.744 UTC [18842] LOG: received smart shutdown request
2026-08-10 23:40:52.748 UTC [18842] LOG: background worker "logical replication launcher" (PID 18858) exited with exit code 1
2026-08-10 23:40:52.748 UTC [18847] LOG: shutting down
2026-08-10 23:40:52.922 UTC [18842] LOG: database system is shut down
2026-08-10T23:40:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54544 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:40:53 dist-test-slave-pww0 krb5kdc[20694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:40:53 dist-test-slave-pww0 krb5kdc[20694](info): set up 2 sockets
Aug 10 23:40:53 dist-test-slave-pww0 krb5kdc[20694](info): commencing operation
krb5kdc: starting...
W20260810 23:40:54.330961 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.002s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:40:54 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:40:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/rangeradmin_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/rangeradmin_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/rangerlookup_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/rangerlookup_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
I20260810 23:40:54.750998 30852 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdkhI7B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/postgres -l logfile start
2026-08-10 23:40:57.227 UTC [20761] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:40:57.227 UTC [20761] LOG: listening on IPv4 address "127.30.33.20", port 49343
2026-08-10 23:40:57.230 UTC [20761] LOG: listening on Unix socket "/tmp/.s.PGSQL.49343"
2026-08-10 23:40:57.254 UTC [20763] LOG: database system was shut down at 2026-08-10 23:40:55 UTC
2026-08-10 23:40:57.257 UTC [20761] LOG: database system is ready to accept connections
I20260810 23:40:58.404107 30852 mini_postgres.cc:96] Postgres bound to 49343
2026-08-10 23:40:58.428 UTC [20781] FATAL: database "slave" does not exist
127.30.33.20:49343 - accepting connections
I20260810 23:40:58.429461 30852 mini_ranger.cc:161] Starting Ranger...
I20260810 23:40:58.462962 30852 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:40:58.958626 30852 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:40:58.958745 30852 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-admin
2026-08-10 23:40:59,276 [I] DB FLAVOR :POSTGRES
2026-08-10 23:40:59,276 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:40:59,276 [I] Checking connection..
2026-08-10 23:40:59,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:40:59,538 [I] Checking connection passed.
2026-08-10 23:40:59,538 [I] --------- Verifying version history table ---------
2026-08-10 23:40:59,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:40:59,799 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:40:59,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:41:00,081 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:41:00,081 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:41:00,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:41:00.343 UTC [20902] WARNING: there is no transaction in progress
2026-08-10 23:41:00,352 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:41:00,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:41:00,635 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:41:00,636 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:41:00,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:41:00,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:41:01,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pww0.c.gcp-upstream.internal','N') ;"
2026-08-10 23:41:01,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:41:01,707 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:41:01,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:41:01,963 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:41:01,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:41:02,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:41:02,507 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:41:02,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:41:03.506 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.524 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.541 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.556 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.576 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.593 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.651 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.659 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.675 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.688 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.700 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.715 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.726 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.733 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.741 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:03.748 UTC [21107] WARNING: there is no transaction in progress
2026-08-10 23:41:04,433 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:41:04,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:41:04,677 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:41:04,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:41:04,923 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:41:04,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:41:05,206 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:41:05,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:41:05,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:41:05,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pww0.c.gcp-upstream.internal';"
2026-08-10 23:41:06,008 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:41:06,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:41:06,270 [I] DB_PATCHES have already been applied
I20260810 23:41:06.276358 30852 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdkhI7B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:41:06.276456 30852 mini_ranger.cc:193] Using host: 127.30.33.20
I20260810 23:41:06.297230 30852 mini_ranger.cc:227] Ranger admin URL: http://127.30.33.20:42689
Aug 10, 2026 11:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:41:07.661402 30852 mini_ranger.cc:160] Time spent starting Ranger: real 9.232s user 0.000s sys 0.069s
Aug 10, 2026 11:41:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:41:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:41:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:41:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:41:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:41:30.478210 30852 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/rangerkms_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/rangerkms_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.33.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.33.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.33.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/HTTP_127.30.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:41:30 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405290, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:41:30.692296 30852 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:41:30.720685 30852 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:41:31.588671 30852 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:41:31.588822 30852 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms
I20260810 23:41:31.588860 30852 mini_ranger_kms.cc:227] Using postgres at 127.30.33.20:49343
2026-08-10 23:41:31,940 [I] DB FLAVOR :POSTGRES
2026-08-10 23:41:31,940 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:41:31,940 [I] Checking connection
2026-08-10 23:41:31,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:41:32,198 [I] connection success
2026-08-10 23:41:32,199 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:41:32,199 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:41:32,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:41:32,462 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:41:32,462 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:41:32,462 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:41:32,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.33.20:49343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:41:32,753 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:41:33.273813 30852 mini_ranger_kms.cc:310] Created kms service
I20260810 23:41:33.477101 30852 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:41:33.524525 30852 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:41:33 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405290, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:33.952966 30852 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:41:33.953307 30852 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdkhI7B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdkhI7B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:41:33.953346 30852 mini_ranger_kms.cc:240] Using host: 127.30.33.20
I20260810 23:41:33.981665 30852 mini_ranger_kms.cc:276] Ranger KMS PID: 21436
I20260810 23:41:33.981817 30852 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.33.20:47839
Aug 10, 2026 11:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:41:36 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786405296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.33.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 23:41:36 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.33.20@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:36.785151 30852 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.093s user 0.051s sys 0.344s
I20260810 23:41:36.785436 30852 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:41:36.785621 30852 mini_ranger_kms.cc:401] 127.30.33.20:47839/kms/v1/keys
Aug 10 23:41:37 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.33.20@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
Aug 10 23:41:39 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:41:39.584504 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:32879
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:32879
--ranger_config_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:39.929509 21546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:39.929857 21546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:39.929996 21546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:39.937759 21546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:41:39.937903 21546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:39.937985 21546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:39.938064 21546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:41:39.938131 21546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:41:39.948663 21546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.33.20:47839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:32879
--ranger_config_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:39.950035 21546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:39.951438 21546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:39.964012 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:39.964088 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:39.964088 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:39.964190 21546 server_base.cc:1034] running on GCE node
I20260810 23:41:39.965502 21546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:39.966995 21546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:39.968206 21546 hybrid_clock.cc:648] HybridClock initialized: now 1786405299968150 us; error 54 us; skew 500 ppm
I20260810 23:41:39.968571 21546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:39 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405299, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:39.971809 21546 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:41:39.973359 21546 webserver.cc:460] Webserver started at http://127.30.33.62:42033/ using document root <none> and password file <none>
I20260810 23:41:39.974025 21546 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:39.974138 21546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:39.974453 21546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:41:39 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405299, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:40.034149 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "ac7ad679dd274b2ba16451ecc982317b"
format_stamp: "Formatted at 2026-08-10 23:41:40 on dist-test-slave-pww0"
server_key: "b662cf4782a085000963d6e37cf18975"
server_key_iv: "5f03c0c9f999b466c50cb12240efd620"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:40.035084 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "ac7ad679dd274b2ba16451ecc982317b"
format_stamp: "Formatted at 2026-08-10 23:41:40 on dist-test-slave-pww0"
server_key: "b662cf4782a085000963d6e37cf18975"
server_key_iv: "5f03c0c9f999b466c50cb12240efd620"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:40.041271 21546 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:41:40.064136 21579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:40.066078 21546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 23:41:40.066313 21546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "ac7ad679dd274b2ba16451ecc982317b"
format_stamp: "Formatted at 2026-08-10 23:41:40 on dist-test-slave-pww0"
server_key: "b662cf4782a085000963d6e37cf18975"
server_key_iv: "5f03c0c9f999b466c50cb12240efd620"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:40.066614 21546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:40.081525 21546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:40.085585 21546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:40.085884 21546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:40.103721 21546 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:32879
I20260810 23:41:40.103744 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:32879 every 8 connection(s)
I20260810 23:41:40.105207 21546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:41:40.106071 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 21546
I20260810 23:41:40.106328 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:41:40.107088 30852 external_mini_cluster.cc:1428] Setting key 9c48e56da88aaf2a2349fcc956dba35f
I20260810 23:41:40.109848 21681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:41:40 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:40.123072 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Bootstrap starting.
I20260810 23:41:40.126587 21681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:40.127786 21681 log.cc:824] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:40.130843 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:40.109236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.30.33.62:32879)
0810 23:41:40.110025 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:40.110038 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:40.110083 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:41:40.111788 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:40.111812 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:40.112244 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:40.112611 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:40.114367 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:40.115818 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:40.117650 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:40.118411 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:40.118546 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:40.123188 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:40.123197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:40.123230 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:40.123287 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:40.126381 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:40.127249 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:40.127257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:40.127265 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:40.127452 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:40.127951 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:40.127956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:40.127959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:40.128555 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:40.128873 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:40.129388 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 23:41:40.129818 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":367,"threads_started":1}
I20260810 23:41:40.130923 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: No bootstrap required, opened a new log
I20260810 23:41:40.139246 21681 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:40.139851 21681 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:40.139973 21681 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7ad679dd274b2ba16451ecc982317b, State: Initialized, Role: FOLLOWER
I20260810 23:41:40.140568 21681 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:40.140787 21681 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:41:40.140887 21681 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:41:40.141013 21681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:40.143199 21681 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:40.143893 21681 leader_election.cc:302] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac7ad679dd274b2ba16451ecc982317b; no voters:
I20260810 23:41:40.144595 21681 leader_election.cc:288] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:41:40.144691 21689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:41:40.145430 21689 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 LEADER]: Becoming Leader. State: Replica: ac7ad679dd274b2ba16451ecc982317b, State: Running, Role: LEADER
I20260810 23:41:40.145969 21689 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:40.146729 21681 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:41:40.149053 21695 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac7ad679dd274b2ba16451ecc982317b. Latest consensus state: current_term: 1 leader_uuid: "ac7ad679dd274b2ba16451ecc982317b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } } }
I20260810 23:41:40.149025 21693 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac7ad679dd274b2ba16451ecc982317b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } } }
I20260810 23:41:40.149505 21693 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:40.149506 21695 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:40.155611 21681 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:41:40.156041 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:41:40.168231 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:41:40.180877 21704 catalog_manager.cc:1347] Generated new cluster ID: 6921003346524a9cb8fa829841fe223d
I20260810 23:41:40.181016 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:41:40.190004 21704 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:41:40.191506 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:41:40.199257 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Generated new TSK 0
I20260810 23:41:40.200182 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:41:40.985882 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:32879
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:41:41 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786405301, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:41:41.479820 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:41.480329 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:41.480516 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:41.490612 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:41.490747 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:41.490901 21764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:41:41.501539 21764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.33.20:47839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:41.502938 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:41.504352 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:41.533605 21784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:41.533682 21785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:41.542150 21794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:41.542253 21764 server_base.cc:1034] running on GCE node
I20260810 23:41:41.543248 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:41.544513 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:41.545842 21764 hybrid_clock.cc:648] HybridClock initialized: now 1786405301545758 us; error 88 us; skew 500 ppm
I20260810 23:41:41.546202 21764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:41 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405301, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:41.550734 21764 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:41:41.552850 21764 webserver.cc:460] Webserver started at http://127.30.33.1:46685/ using document root <none> and password file <none>
I20260810 23:41:41.553704 21764 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:41.553841 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:41.554195 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:41:41 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405301, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:41.585803 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "1f21ced16c10489cbae6a4a8192c1763"
format_stamp: "Formatted at 2026-08-10 23:41:41 on dist-test-slave-pww0"
server_key: "e7a9f04b013887f36b0e7a3d46dc6520"
server_key_iv: "61ab9bc7dc9da5160484dfa91fd87227"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:41.588047 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "1f21ced16c10489cbae6a4a8192c1763"
format_stamp: "Formatted at 2026-08-10 23:41:41 on dist-test-slave-pww0"
server_key: "e7a9f04b013887f36b0e7a3d46dc6520"
server_key_iv: "61ab9bc7dc9da5160484dfa91fd87227"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:41.598763 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.000s
I20260810 23:41:41.625648 21803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:41.627852 21764 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260810 23:41:41.628124 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "1f21ced16c10489cbae6a4a8192c1763"
format_stamp: "Formatted at 2026-08-10 23:41:41 on dist-test-slave-pww0"
server_key: "e7a9f04b013887f36b0e7a3d46dc6520"
server_key_iv: "61ab9bc7dc9da5160484dfa91fd87227"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:41.628425 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:41.691282 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:41.697053 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:41.697504 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:41.699536 21764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:41.701467 21764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:41.701573 21764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:41.701719 21764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:41.701784 21764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:41:41 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405301, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:42.058091 22005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:34769 every 8 connection(s)
I20260810 23:41:42.058467 21764 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:34769
I20260810 23:41:42.060279 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:41:42.065651 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 21764
I20260810 23:41:42.065912 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:41:42.066799 30852 external_mini_cluster.cc:1428] Setting key cd83da612b12add9412450176cf64f0a
Aug 10 23:41:42 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405301, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:42.114929 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:42.068186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:40077 (local address 127.30.33.62:32879)
0810 23:41:42.083000 (+ 14814us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:42.083007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:42.083035 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:41:42.085787 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:42.085792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:42.085904 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:42.086007 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:42.088483 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.089667 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.092556 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.093202 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.093321 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:42.098154 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:42.098163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:42.098168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:42.098216 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:42.101752 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:42.102867 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:42.102873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:42.102877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:42.103056 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:42.112578 (+ 9522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:42.112586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:42.112589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:42.113019 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:42.113171 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:42.114294 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0810 23:41:42.114553 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14670,"thread_start_us":15269,"threads_started":1}
I20260810 23:41:42.117496 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:42.068429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:32879 (local address 127.30.33.1:40077)
0810 23:41:42.080963 (+ 12534us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:42.081028 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:42.085225 (+ 4197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:42.086484 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:42.086497 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:42.087188 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:42.088197 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:42.088246 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.089935 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.089941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:42.092313 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:42.092338 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.093754 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.093762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:42.094068 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:42.094912 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:42.094945 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:42.097811 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:42.102008 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:42.102020 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:42.102045 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:42.102697 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:42.111953 (+ 9256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:42.111959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:42.111962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:42.112358 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:42.113304 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:42.113314 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:42.113830 (+ 516us) client_negotiation.cc:769] Sending connection context
0810 23:41:42.115435 (+ 1605us) client_negotiation.cc:240] Negotiation successful
0810 23:41:42.116080 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12125,"thread_start_us":15569,"threads_started":1}
I20260810 23:41:42.120129 22006 heartbeater.cc:347] Connected to a master server at 127.30.33.62:32879
I20260810 23:41:42.120478 22006 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:42.121398 22006 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
I20260810 23:41:42.124234 21609 ts_manager.cc:194] Registered new tserver with Master: 1f21ced16c10489cbae6a4a8192c1763 (127.30.33.1:34769)
I20260810 23:41:42.126233 21609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:40077
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Aug 10 23:41:42 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405301, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
W20260810 23:41:42.201916 30852 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.30.33.2: real 0.132s user 0.000s sys 0.027s
I20260810 23:41:42.202883 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:32879
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:42.699239 22022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:42.699606 22022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:42.699762 22022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:42.707253 22022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:42.707388 22022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:42.707572 22022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:41:42.717935 22022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.33.20:47839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:42.719334 22022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:42.720744 22022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:42.735687 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:42.736187 22041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:42.736467 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:42.736851 22022 server_base.cc:1034] running on GCE node
I20260810 23:41:42.737641 22022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:42.738904 22022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:42.740142 22022 hybrid_clock.cc:648] HybridClock initialized: now 1786405302740067 us; error 64 us; skew 500 ppm
I20260810 23:41:42.740549 22022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:42 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405302, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:42.744694 22022 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:41:42.746580 22022 webserver.cc:460] Webserver started at http://127.30.33.2:40159/ using document root <none> and password file <none>
I20260810 23:41:42.747577 22022 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:42.747715 22022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:42.748064 22022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:41:42 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405302, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:42.768016 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "bb9e9ee37f1344b0976e7a34c1d0579b"
format_stamp: "Formatted at 2026-08-10 23:41:42 on dist-test-slave-pww0"
server_key: "dd3b9ceaee9a3773926febe86f60a58f"
server_key_iv: "9c422dfe0526c2c62c2e05d9d7525e8d"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:42.768955 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "bb9e9ee37f1344b0976e7a34c1d0579b"
format_stamp: "Formatted at 2026-08-10 23:41:42 on dist-test-slave-pww0"
server_key: "dd3b9ceaee9a3773926febe86f60a58f"
server_key_iv: "9c422dfe0526c2c62c2e05d9d7525e8d"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:42.776132 22022 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 23:41:42.795439 22055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:42.797617 22022 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:41:42.797842 22022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "bb9e9ee37f1344b0976e7a34c1d0579b"
format_stamp: "Formatted at 2026-08-10 23:41:42 on dist-test-slave-pww0"
server_key: "dd3b9ceaee9a3773926febe86f60a58f"
server_key_iv: "9c422dfe0526c2c62c2e05d9d7525e8d"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:42.798116 22022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:42.828939 22022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:42.835100 22022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:42.835493 22022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:42.836827 22022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:42.838781 22022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:42.838889 22022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:42.839071 22022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:42.839149 22022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:42.884282 22022 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:37755
I20260810 23:41:42.884333 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:37755 every 8 connection(s)
I20260810 23:41:42.886233 22022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:41:42.887632 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 22022
I20260810 23:41:42.887810 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:41:42.888428 30852 external_mini_cluster.cc:1428] Setting key f711b6c0c4b01d59b845c1c2454a8fa5
Aug 10 23:41:42 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405302, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:42.933209 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:42.897270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:51941 (local address 127.30.33.62:32879)
0810 23:41:42.907998 (+ 10728us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:42.908005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:42.908703 (+ 698us) server_negotiation.cc:407] Connection header received
0810 23:41:42.911611 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:42.911618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:42.911721 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:42.911812 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:42.914311 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.915543 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.918338 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.918998 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.919118 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:42.924301 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:42.924309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:42.924314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:42.924364 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:42.928649 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:42.929870 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:42.929877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:42.929881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:42.930053 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:42.930792 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:42.930798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:42.930801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:42.931222 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:42.931361 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:42.932521 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0810 23:41:42.932815 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10556,"thread_start_us":10379,"threads_started":1}
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
I20260810 23:41:42.936178 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:42.898049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:32879 (local address 127.30.33.2:51941)
0810 23:41:42.908480 (+ 10431us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:42.908547 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:42.911154 (+ 2607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:42.912459 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:42.912473 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:42.913142 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:42.914051 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:42.914081 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.915835 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.915842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:42.918117 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:42.918137 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:42.919586 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:42.919596 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:42.919974 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:42.920916 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:42.920960 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:42.923950 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:42.928931 (+ 4981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:42.928942 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:42.928963 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:42.929691 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:42.930233 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:42.930237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:42.930240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:42.930634 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:42.931524 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:42.931533 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:42.932045 (+ 512us) client_negotiation.cc:769] Sending connection context
0810 23:41:42.933811 (+ 1766us) client_negotiation.cc:240] Negotiation successful
0810 23:41:42.934446 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10064,"thread_start_us":9781,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
I20260810 23:41:42.938141 22271 heartbeater.cc:347] Connected to a master server at 127.30.33.62:32879
I20260810 23:41:42.938517 22271 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:42.939442 22271 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
I20260810 23:41:42.941444 21609 ts_manager.cc:194] Registered new tserver with Master: bb9e9ee37f1344b0976e7a34c1d0579b (127.30.33.2:37755)
I20260810 23:41:42.942651 21609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:51941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:41:42.978850 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:32879
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:41:43.130591 22006 heartbeater.cc:503] Master 127.30.33.62:32879 was elected leader, sending a full tablet report...
W20260810 23:41:43.414546 22284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:43.415030 22284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:43.415153 22284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:43.422945 22284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:43.423079 22284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:43.423194 22284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:41:43.433602 22284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.33.20:47839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:43.435096 22284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:43.436530 22284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:43.450302 22293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:43.450270 22294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:43.450217 22298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:43.451327 22284 server_base.cc:1034] running on GCE node
I20260810 23:41:43.452019 22284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:43.453125 22284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:43.454375 22284 hybrid_clock.cc:648] HybridClock initialized: now 1786405303454314 us; error 52 us; skew 500 ppm
I20260810 23:41:43.454759 22284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:43 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405303, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:43.458217 22284 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:41:43.459939 22284 webserver.cc:460] Webserver started at http://127.30.33.3:42057/ using document root <none> and password file <none>
I20260810 23:41:43.460695 22284 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:43.460822 22284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:43.461155 22284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:41:43 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405303, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:43.476467 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "2926dc869d0c415cb59c20d54553f64e"
format_stamp: "Formatted at 2026-08-10 23:41:43 on dist-test-slave-pww0"
server_key: "90988c21b2f112156d30d11fc9edd47c"
server_key_iv: "43eed3b06859814f32502d07a0390164"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:43.477334 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "2926dc869d0c415cb59c20d54553f64e"
format_stamp: "Formatted at 2026-08-10 23:41:43 on dist-test-slave-pww0"
server_key: "90988c21b2f112156d30d11fc9edd47c"
server_key_iv: "43eed3b06859814f32502d07a0390164"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:43.482966 22284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:41:43.497908 22307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:43.499789 22284 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:41:43.500003 22284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "2926dc869d0c415cb59c20d54553f64e"
format_stamp: "Formatted at 2026-08-10 23:41:43 on dist-test-slave-pww0"
server_key: "90988c21b2f112156d30d11fc9edd47c"
server_key_iv: "43eed3b06859814f32502d07a0390164"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:43.500259 22284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:43.522143 22284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:43.526929 22284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:43.527261 22284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:43.528335 22284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:43.529887 22284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:43.529973 22284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:43.530102 22284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:43.530155 22284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:43.560969 22284 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:41401
I20260810 23:41:43.561007 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:41401 every 8 connection(s)
I20260810 23:41:43.562330 22284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:41:43.563377 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 22284
I20260810 23:41:43.563561 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:41:43.564064 30852 external_mini_cluster.cc:1428] Setting key bab2a60b98db383f471afb35e3c7fe56
Aug 10 23:41:43 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405303, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:43.583606 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:43.565455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:37447 (local address 127.30.33.62:32879)
0810 23:41:43.565976 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:43.565982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:43.567240 (+ 1258us) server_negotiation.cc:407] Connection header received
0810 23:41:43.569145 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:43.569149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:43.569238 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:43.569332 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:43.570964 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.571813 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.573519 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.574007 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.574110 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:43.577498 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:43.577503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:43.577506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:43.577541 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:43.580771 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.581579 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.581584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.581586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.581720 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.582076 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.582079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.582081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.582371 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:43.582480 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:43.583111 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 23:41:43.583297 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":234,"threads_started":1}
I20260810 23:41:43.584690 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:43.566102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:32879 (local address 127.30.33.3:37447)
0810 23:41:43.567053 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:43.567111 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:43.568883 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:43.569615 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:43.569624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:43.570076 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:43.570769 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:43.570791 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.572018 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.572022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:43.573372 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:43.573387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.574230 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.574236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:43.574457 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:43.575102 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:43.575125 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:43.577249 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:43.580946 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:43.580954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:43.580970 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:43.581466 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:43.581803 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:43.581807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:43.581809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:43.581999 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:43.582458 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:43.582464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:43.582818 (+ 354us) client_negotiation.cc:769] Sending connection context
0810 23:41:43.583051 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 23:41:43.583500 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":396,"threads_started":1}
I20260810 23:41:43.586017 22508 heartbeater.cc:347] Connected to a master server at 127.30.33.62:32879
I20260810 23:41:43.586301 22508 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:43.586966 22508 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
I20260810 23:41:43.588413 21609 ts_manager.cc:194] Registered new tserver with Master: 2926dc869d0c415cb59c20d54553f64e (127.30.33.3:41401)
I20260810 23:41:43.589231 21609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:37447
I20260810 23:41:43.592904 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:41:43.594578 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 21764
Aug 10 23:41:43 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:41:43.611353 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:43.595536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.30.33.1:34769)
0810 23:41:43.596106 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:43.596113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:43.596143 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:41:43.596239 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:43.596246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:43.596458 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:43.596590 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:43.598067 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.599107 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.600679 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.601255 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.601358 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:43.605770 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:43.605784 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:43.605799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:43.605839 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:43.608499 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.609296 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.609304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.609306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.609453 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.609864 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.609867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.609869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.610207 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:43.610472 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:43.610778 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 23:41:43.610996 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260810 23:41:43.612679 21998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:43.696275 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 21764
I20260810 23:41:43.725232 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 22022
Aug 10 23:41:43 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:41:43.742872 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:43.726141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.30.33.2:37755)
0810 23:41:43.726666 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:43.726674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:43.726701 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:41:43.726763 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:43.726769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:43.726994 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:43.727155 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:43.728586 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.729576 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.731246 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.731953 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.732076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:43.736610 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:43.736626 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:43.736651 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:43.736703 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:43.739649 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.740514 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.740523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.740525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.740696 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.741127 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.741131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.741133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.741532 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:43.741869 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:43.742199 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:41:43.742489 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":243,"threads_started":1}
I20260810 23:41:43.744355 22265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:43.830412 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 22022
I20260810 23:41:43.856387 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 22284
Aug 10 23:41:43 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:41:43.873148 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:43.857512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.30.33.3:41401)
0810 23:41:43.858201 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:43.858208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:43.858251 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:41:43.858315 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:43.858322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:43.858540 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:43.858665 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:43.860080 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.861144 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.862489 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:43.863057 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:43.863154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:43.867655 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:43.867670 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:43.867687 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:43.867733 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:43.870413 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.871160 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.871167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.871169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.871316 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:43.871688 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:43.871692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:43.871694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:43.872048 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:43.872339 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:43.872534 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 23:41:43.872795 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":368,"threads_started":1}
I20260810 23:41:43.874477 22502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:43.958422 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 22284
I20260810 23:41:43.982223 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 21546
I20260810 23:41:43.983525 21667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:44.090972 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 21546
I20260810 23:41:44.122581 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:32879
--webserver_interface=127.30.33.62
--webserver_port=42033
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:32879
--ranger_config_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:44.467272 22533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:44.467619 22533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:44.467754 22533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:44.475170 22533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:41:44.475297 22533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:44.475344 22533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:44.475421 22533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:41:44.475541 22533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:41:44.486018 22533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:32879
--ranger_config_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=42033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:44.487493 22533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:44.488955 22533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:44.501528 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:44.501585 22546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:44.501478 22542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:44.502564 22533 server_base.cc:1034] running on GCE node
I20260810 23:41:44.503399 22533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:44.504981 22533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:44.506228 22533 hybrid_clock.cc:648] HybridClock initialized: now 1786405304506181 us; error 43 us; skew 500 ppm
I20260810 23:41:44.506626 22533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:44 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405304, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:44.522670 22533 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:41:44.524431 22533 webserver.cc:460] Webserver started at http://127.30.33.62:42033/ using document root <none> and password file <none>
I20260810 23:41:44.525123 22533 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:44.525269 22533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:41:44 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405304, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:44.602988 22533 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:41:44.606336 22565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:44.608018 22533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:41:44.608255 22533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "ac7ad679dd274b2ba16451ecc982317b"
format_stamp: "Formatted at 2026-08-10 23:41:40 on dist-test-slave-pww0"
server_key: "b662cf4782a085000963d6e37cf18975"
server_key_iv: "5f03c0c9f999b466c50cb12240efd620"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:44.609267 22533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:44.625345 22533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:44.629709 22533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:44.630007 22533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:44.648725 22533 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:32879
I20260810 23:41:44.648746 22657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:32879 every 8 connection(s)
I20260810 23:41:44.650373 22533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:41:44.654379 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 22533
I20260810 23:41:44.655428 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:34769
--local_ip_for_outbound_sockets=127.30.33.1
--tserver_master_addrs=127.30.33.62:32879
--webserver_port=46685
--webserver_interface=127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:41:44.659039 22659 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:41:44.663803 22659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Bootstrap starting.
I20260810 23:41:44.683911 22659 log.cc:824] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:44.694046 22659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:41:44.694592 22659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Bootstrap complete.
I20260810 23:41:44.703974 22659 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:44.704878 22659 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7ad679dd274b2ba16451ecc982317b, State: Initialized, Role: FOLLOWER
I20260810 23:41:44.705557 22659 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:44.705780 22659 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:41:44.705894 22659 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:41:44.706055 22659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:41:44.710153 22659 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:44.710686 22659 leader_election.cc:302] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac7ad679dd274b2ba16451ecc982317b; no voters:
I20260810 23:41:44.711313 22659 leader_election.cc:288] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:41:44.711485 22664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:41:44.712697 22664 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [term 2 LEADER]: Becoming Leader. State: Replica: ac7ad679dd274b2ba16451ecc982317b, State: Running, Role: LEADER
I20260810 23:41:44.713311 22664 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } }
I20260810 23:41:44.713717 22659 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:41:44.715932 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac7ad679dd274b2ba16451ecc982317b. Latest consensus state: current_term: 2 leader_uuid: "ac7ad679dd274b2ba16451ecc982317b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } } }
I20260810 23:41:44.716361 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:44.716895 22665 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac7ad679dd274b2ba16451ecc982317b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7ad679dd274b2ba16451ecc982317b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 32879 } } }
I20260810 23:41:44.717163 22665 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:44.719434 22676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:41:44.728107 22676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:41:44.729771 22676 catalog_manager.cc:1259] Loaded cluster ID: 6921003346524a9cb8fa829841fe223d
I20260810 23:41:44.730011 22676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:41:44.734715 22676 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:41:44.736896 22676 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Loaded TSK: 0
I20260810 23:41:44.738787 22676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:41:44.742741 22659 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 23:41:45.074406 22662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:45.074808 22662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:45.074954 22662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:45.086187 22662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:45.086349 22662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:45.086541 22662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:41:45.100857 22662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:34769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=46685
--tserver_master_addrs=127.30.33.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:45.102581 22662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:45.104544 22662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:45.161190 22730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:45.165771 22729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:45.165993 22662 server_base.cc:1034] running on GCE node
W20260810 23:41:45.166571 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:45.167520 22662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:45.168956 22662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:45.170971 22662 hybrid_clock.cc:648] HybridClock initialized: now 1786405305170889 us; error 70 us; skew 500 ppm
I20260810 23:41:45.171411 22662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:45 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405305, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:45.176203 22662 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:41:45.178792 22662 webserver.cc:460] Webserver started at http://127.30.33.1:46685/ using document root <none> and password file <none>
I20260810 23:41:45.179872 22662 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:45.180058 22662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:41:45 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405305, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:45.225507 22662 fs_manager.cc:709] Time spent opening directory manager: real 0.024s user 0.009s sys 0.009s
I20260810 23:41:45.229928 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:45.232461 22662 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 23:41:45.232705 22662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "1f21ced16c10489cbae6a4a8192c1763"
format_stamp: "Formatted at 2026-08-10 23:41:41 on dist-test-slave-pww0"
server_key: "e7a9f04b013887f36b0e7a3d46dc6520"
server_key_iv: "61ab9bc7dc9da5160484dfa91fd87227"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:45.234166 22662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:45.291731 22662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:45.296470 22662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:45.296769 22662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:45.298013 22662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:45.301175 22662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:45.301313 22662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:45.301430 22662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:45.301493 22662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:45.407843 22662 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:34769
I20260810 23:41:45.407878 22956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:34769 every 8 connection(s)
I20260810 23:41:45.409441 22662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:41:45.417627 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 22662
I20260810 23:41:45.418357 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:37755
--local_ip_for_outbound_sockets=127.30.33.2
--tserver_master_addrs=127.30.33.62:32879
--webserver_port=40159
--webserver_interface=127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:41:45 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405305, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:45.464295 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:45.415624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:36141 (local address 127.30.33.62:32879)
0810 23:41:45.426059 (+ 10435us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:45.426087 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:45.435056 (+ 8969us) server_negotiation.cc:407] Connection header received
0810 23:41:45.435721 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:45.435760 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:45.436414 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:45.436952 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:45.439792 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:45.441932 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:45.444548 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:45.445692 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:45.445888 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:45.452803 (+ 6915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:45.452820 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:45.452847 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:45.452914 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:45.457200 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:45.458412 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:45.458422 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:45.458432 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:45.458674 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:45.459320 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:45.459327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:45.459330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:45.460173 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:45.460411 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:45.462116 (+ 1705us) server_negotiation.cc:299] Negotiation successful
0810 23:41:45.462571 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10097,"thread_start_us":9791,"threads_started":1}
I20260810 23:41:45.466473 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:45.414150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:32879 (local address 127.30.33.1:36141)
0810 23:41:45.431789 (+ 17639us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:45.431868 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:45.434518 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:45.437847 (+ 3329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:45.437860 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:45.438594 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:45.439497 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:45.439533 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:45.442192 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:45.442197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:45.444344 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:45.444363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:45.446426 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:45.446436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:45.446740 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:45.447622 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:45.447700 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:45.452342 (+ 4642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:45.457450 (+ 5108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:45.457462 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:45.457481 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:45.458197 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:45.458851 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:45.458858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:45.458862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:45.459141 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:45.460507 (+ 1366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:45.460516 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:45.461017 (+ 501us) client_negotiation.cc:769] Sending connection context
0810 23:41:45.461364 (+ 347us) client_negotiation.cc:240] Negotiation successful
0810 23:41:45.465140 (+ 3776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12282,"thread_start_us":12000,"threads_started":1}
I20260810 23:41:45.470690 22958 heartbeater.cc:347] Connected to a master server at 127.30.33.62:32879
I20260810 23:41:45.471074 22958 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:45.472081 22958 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
W20260810 23:41:45.475935 22958 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:32879 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:41:45.476189 22958 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:45.476500 22958 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
I20260810 23:41:45.477949 22958 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:45.478245 22958 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
W20260810 23:41:45.479430 22958 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:41:45.958014 22965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:45.958348 22965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:45.958482 22965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:45.966601 22965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:45.966763 22965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:45.967012 22965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:41:45.979846 22965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:37755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=40159
--tserver_master_addrs=127.30.33.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:45.981853 22965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:45.983704 22965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:46.007513 22995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:46.008566 22996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:46.010353 22998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:46.010651 22965 server_base.cc:1034] running on GCE node
I20260810 23:41:46.011572 22965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:46.012794 22965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:46.014027 22965 hybrid_clock.cc:648] HybridClock initialized: now 1786405306013948 us; error 72 us; skew 500 ppm
I20260810 23:41:46.014452 22965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:46.018020 22965 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:41:46.020771 22965 webserver.cc:460] Webserver started at http://127.30.33.2:40159/ using document root <none> and password file <none>
I20260810 23:41:46.021538 22965 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:46.021687 22965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:46.054613 22965 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.002s
I20260810 23:41:46.072256 23009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:46.074544 22965 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.000s sys 0.016s
I20260810 23:41:46.074892 22965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "bb9e9ee37f1344b0976e7a34c1d0579b"
format_stamp: "Formatted at 2026-08-10 23:41:42 on dist-test-slave-pww0"
server_key: "dd3b9ceaee9a3773926febe86f60a58f"
server_key_iv: "9c422dfe0526c2c62c2e05d9d7525e8d"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:46.076601 22965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:46.102975 22965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:46.108923 22965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:46.109345 22965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:46.117991 22965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:46.121313 22965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:46.121515 22965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 23:41:46.121701 22965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:46.121812 22965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:46.163398 22965 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:37755
I20260810 23:41:46.163437 23231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:37755 every 8 connection(s)
I20260810 23:41:46.165124 22965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:41:46.171260 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 22965
I20260810 23:41:46.172153 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.33.20:47839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:41401
--local_ip_for_outbound_sockets=127.30.33.3
--tserver_master_addrs=127.30.33.62:32879
--webserver_port=42057
--webserver_interface=127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:46.219131 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:46.170001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:58493 (local address 127.30.33.62:32879)
0810 23:41:46.192500 (+ 22499us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:46.192507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:46.194010 (+ 1503us) server_negotiation.cc:407] Connection header received
0810 23:41:46.197909 (+ 3899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:46.197916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:46.198029 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:46.198127 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:46.201246 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.202415 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.205054 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.205705 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.205836 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:46.210776 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:46.210784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:46.210789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:46.210837 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:46.214672 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.215920 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.215926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.215930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.216102 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.216685 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.216690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.216694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.217131 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:46.217272 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:46.218442 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0810 23:41:46.218714 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22345,"thread_start_us":22086,"threads_started":1}
I20260810 23:41:46.221738 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:46.169245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:32879 (local address 127.30.33.2:58493)
0810 23:41:46.193179 (+ 23934us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:46.193251 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:46.197387 (+ 4136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:46.198811 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:46.198825 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:46.200038 (+ 1213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:46.200991 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:46.201022 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.202650 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.202655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:46.204814 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:46.204845 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.206274 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.206282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:46.206595 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:46.207483 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:46.207521 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:46.210429 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:46.215004 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:46.215015 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:46.215036 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:46.215752 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:46.216265 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:46.216271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:46.216274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:46.216550 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:46.217416 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:46.217426 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:46.217923 (+ 497us) client_negotiation.cc:769] Sending connection context
0810 23:41:46.219493 (+ 1570us) client_negotiation.cc:240] Negotiation successful
0810 23:41:46.220172 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23537,"thread_start_us":24644,"threads_started":1}
I20260810 23:41:46.224236 23232 heartbeater.cc:347] Connected to a master server at 127.30.33.62:32879
I20260810 23:41:46.224609 23232 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:46.225633 23232 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
I20260810 23:41:46.228293 22600 ts_manager.cc:194] Registered new tserver with Master: bb9e9ee37f1344b0976e7a34c1d0579b (127.30.33.2:37755)
I20260810 23:41:46.230626 22600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:58493
I20260810 23:41:46.479635 22958 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:46.480183 22958 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
I20260810 23:41:46.481703 22600 ts_manager.cc:194] Registered new tserver with Master: 1f21ced16c10489cbae6a4a8192c1763 (127.30.33.1:34769)
I20260810 23:41:46.482925 22600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:36141
W20260810 23:41:46.629096 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:46.629565 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:46.629738 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
W20260810 23:41:46.639567 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:46.639783 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:46.640004 23240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:41:46.659993 23240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.33.20:47839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:41401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=42057
--tserver_master_addrs=127.30.33.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:46.662184 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:46.666875 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:46.715512 23251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:46.720324 23250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:46.751591 23240 server_base.cc:1034] running on GCE node
W20260810 23:41:46.752785 23254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:46.753640 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:46.755745 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:46.757025 23240 hybrid_clock.cc:648] HybridClock initialized: now 1786405306756966 us; error 64 us; skew 500 ppm
I20260810 23:41:46.757403 23240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:46.760792 23240 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:41:46.762787 23240 webserver.cc:460] Webserver started at http://127.30.33.3:42057/ using document root <none> and password file <none>
I20260810 23:41:46.763629 23240 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:46.763782 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:46.786547 23240 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:41:46.790755 23267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:46.792856 23240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:41:46.793081 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "2926dc869d0c415cb59c20d54553f64e"
format_stamp: "Formatted at 2026-08-10 23:41:43 on dist-test-slave-pww0"
server_key: "90988c21b2f112156d30d11fc9edd47c"
server_key_iv: "43eed3b06859814f32502d07a0390164"
server_key_version: "kuduclusterkey@0"
I20260810 23:41:46.794611 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:46.816288 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:46.822528 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:46.822870 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:46.824349 23240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:46.826529 23240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:46.826660 23240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:46.826812 23240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:46.826903 23240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:46.878173 23240 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:41401
I20260810 23:41:46.878214 23502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:41401 every 8 connection(s)
I20260810 23:41:46.880353 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:41:46.887225 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 23240
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:46.934331 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:46.883503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:52109 (local address 127.30.33.62:32879)
0810 23:41:46.902387 (+ 18884us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:46.902393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:46.902415 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:41:46.902489 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:46.902495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:46.902609 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:46.902724 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:46.905327 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.906586 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.920808 (+ 14222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.921451 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.921552 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:46.926199 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:46.926207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:46.926211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:46.926284 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:46.930079 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.931216 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.931222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.931230 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.931399 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.932042 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.932047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.932050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.932435 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:46.932564 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:46.933637 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0810 23:41:46.933906 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18724,"thread_start_us":18579,"threads_started":1}
I20260810 23:41:46.935180 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:46.888524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.30.33.62:32879)
0810 23:41:46.907145 (+ 18621us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:46.907150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:46.916156 (+ 9006us) server_negotiation.cc:407] Connection header received
0810 23:41:46.916376 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:46.916382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:46.916524 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:46.916661 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:46.919097 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.920090 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.922138 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.922581 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.922693 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:46.927834 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:46.927843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:46.927846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:46.927902 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:46.931583 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.932631 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.932637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.932641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.932806 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.933347 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.933353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.933357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.933702 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:46.933965 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:46.934582 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 23:41:46.934826 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18483,"thread_start_us":18321,"threads_started":1}
I20260810 23:41:46.938133 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:46.884387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:32879 (local address 127.30.33.3:52109)
0810 23:41:46.888565 (+ 4178us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:46.888628 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:46.901470 (+ 12842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:46.903285 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:46.903301 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:46.904083 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:46.905036 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:46.905067 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.914611 (+ 9544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.914619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:46.920580 (+ 5961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:46.920614 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.921951 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.921959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:46.922291 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:46.923080 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:46.923112 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:46.925894 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:46.930337 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:46.930349 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:46.930371 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:46.931044 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:46.931626 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:46.931631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:46.931634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:46.931905 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:46.932669 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:46.932678 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:46.933142 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 23:41:46.934941 (+ 1799us) client_negotiation.cc:240] Negotiation successful
0810 23:41:46.935843 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3813,"thread_start_us":3570,"threads_started":1}
I20260810 23:41:46.940101 23504 heartbeater.cc:347] Connected to a master server at 127.30.33.62:32879
I20260810 23:41:46.940459 23504 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:46.941461 23504 heartbeater.cc:511] Master 127.30.33.62:32879 requested a full tablet report, sending...
I20260810 23:41:46.943310 22600 ts_manager.cc:194] Registered new tserver with Master: 2926dc869d0c415cb59c20d54553f64e (127.30.33.3:41401)
I20260810 23:41:46.944494 22600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:52109
I20260810 23:41:46.952725 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:41:46.989379 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:46.963775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.30.33.62:32879)
0810 23:41:46.964111 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:46.964118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:46.969196 (+ 5078us) server_negotiation.cc:407] Connection header received
0810 23:41:46.969514 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:46.969522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:46.969648 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:46.969798 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:46.972172 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.973247 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.975450 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:46.976240 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:46.976415 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:46.982160 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:46.982167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:46.982172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:46.982231 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:46.985966 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.987077 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.987084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.987087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.987240 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:46.987759 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:46.987764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:46.987766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:46.988134 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:46.988382 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:46.988800 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 23:41:46.989064 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
Aug 10 23:41:46 dist-test-slave-pww0 krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405306, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for HTTP/127.30.33.20@KRBTEST.COM
I20260810 23:41:46.996083 22600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:41:47.009058 22600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:41:47.118584 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.080079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:37755 (local address 127.0.0.1:34764)
0810 23:41:47.096099 (+ 16020us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:47.096181 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:47.096421 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:47.106384 (+ 9963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:47.106392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:47.106443 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:41:47.106932 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.106948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.112266 (+ 5318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.112273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.115262 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.115285 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.117551 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.117561 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.117794 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.117859 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 23:41:47.117993 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 23:41:47.118214 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15684,"threads_started":1}
I20260810 23:41:47.133741 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.080121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34769 (local address 127.0.0.1:60380)
0810 23:41:47.114198 (+ 34077us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:47.114223 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:47.114408 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:47.117214 (+ 2806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:47.117221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:47.117256 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:41:47.117756 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.117768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.122273 (+ 4505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.122279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.127572 (+ 5293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.127600 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.132840 (+ 5240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.132848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.133044 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.133085 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:41:47.133185 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:41:47.133362 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33907,"thread_start_us":15495,"threads_started":1}
I20260810 23:41:47.134925 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.080527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.30.33.1:34769)
0810 23:41:47.116184 (+ 35657us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.116195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.116234 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:41:47.116353 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.116362 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.116684 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.116936 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:41:47.119977 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.122053 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.128462 (+ 6409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.130060 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.130217 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.130509 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.134400 (+ 3891us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.134599 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34817,"thread_start_us":47230,"threads_started":1}
I20260810 23:41:47.137457 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.080263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.30.33.2:37755)
0810 23:41:47.105269 (+ 25006us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.105280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.105320 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:41:47.105451 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.105460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.105757 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.106024 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:41:47.107388 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.112001 (+ 4613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.115501 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.117001 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.117141 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.117363 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.136878 (+ 19515us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.137100 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24790,"thread_start_us":26826,"threads_started":1}
I20260810 23:41:47.140365 22847 tablet_service.cc:1467] Processing CreateTablet for tablet dc7f8ca2093845c5960b7809e0f58445 (DEFAULT_TABLE table=test-table [id=178716e7121149ae9835c86c501a070c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:41:47.142295 22847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7f8ca2093845c5960b7809e0f58445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:47.141849 23110 tablet_service.cc:1467] Processing CreateTablet for tablet dc7f8ca2093845c5960b7809e0f58445 (DEFAULT_TABLE table=test-table [id=178716e7121149ae9835c86c501a070c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:41:47.143303 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7f8ca2093845c5960b7809e0f58445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:47.155282 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.088107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.30.33.3:41401)
0810 23:41:47.112478 (+ 24371us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.112489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.119209 (+ 6720us) server_negotiation.cc:407] Connection header received
0810 23:41:47.119515 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.119524 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.119813 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.119985 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:41:47.128160 (+ 8175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.130601 (+ 2441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.151769 (+ 21168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.153504 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.153649 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.153915 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.154691 (+ 776us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.154909 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24154,"thread_start_us":24390,"threads_started":1}
I20260810 23:41:47.156035 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.087816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:41401 (local address 127.0.0.1:52774)
0810 23:41:47.119078 (+ 31262us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:47.119098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:47.119317 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:47.120325 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:47.120330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:47.120357 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:41:47.120524 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.120533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.131420 (+ 10887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.131432 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.133433 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.133452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.154171 (+ 20719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.154178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.154423 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.154471 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:41:47.155563 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0810 23:41:47.155744 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31142,"thread_start_us":26687,"threads_started":1}
I20260810 23:41:47.160545 23374 tablet_service.cc:1467] Processing CreateTablet for tablet dc7f8ca2093845c5960b7809e0f58445 (DEFAULT_TABLE table=test-table [id=178716e7121149ae9835c86c501a070c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:41:47.162329 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7f8ca2093845c5960b7809e0f58445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:47.166180 23549 tablet_bootstrap.cc:492] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Bootstrap starting.
I20260810 23:41:47.172565 23549 tablet_bootstrap.cc:654] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:47.174252 23549 log.cc:824] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:47.184518 23549 tablet_bootstrap.cc:492] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: No bootstrap required, opened a new log
I20260810 23:41:47.185235 23549 ts_tablet_manager.cc:1397] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260810 23:41:47.188270 23553 tablet_bootstrap.cc:492] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Bootstrap starting.
I20260810 23:41:47.196306 23555 tablet_bootstrap.cc:492] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Bootstrap starting.
I20260810 23:41:47.196053 23549 raft_consensus.cc:348] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.196197 23553 tablet_bootstrap.cc:654] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:47.196614 23549 raft_consensus.cc:374] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:47.196823 23549 raft_consensus.cc:729] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f21ced16c10489cbae6a4a8192c1763, State: Initialized, Role: FOLLOWER
I20260810 23:41:47.197723 23549 consensus_queue.cc:260] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.198030 23553 log.cc:824] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:47.210410 23555 tablet_bootstrap.cc:654] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:47.212981 22958 heartbeater.cc:503] Master 127.30.33.62:32879 was elected leader, sending a full tablet report...
I20260810 23:41:47.213363 23555 log.cc:824] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:47.219272 23549 ts_tablet_manager.cc:1428] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Time spent starting tablet: real 0.034s user 0.015s sys 0.011s
I20260810 23:41:47.225129 23555 tablet_bootstrap.cc:492] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: No bootstrap required, opened a new log
I20260810 23:41:47.225544 23555 ts_tablet_manager.cc:1397] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20260810 23:41:47.230736 23553 tablet_bootstrap.cc:492] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: No bootstrap required, opened a new log
I20260810 23:41:47.231069 23553 ts_tablet_manager.cc:1397] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.010s
I20260810 23:41:47.236295 23232 heartbeater.cc:503] Master 127.30.33.62:32879 was elected leader, sending a full tablet report...
I20260810 23:41:47.236101 23555 raft_consensus.cc:348] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.236590 23555 raft_consensus.cc:374] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:47.236678 23555 raft_consensus.cc:729] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2926dc869d0c415cb59c20d54553f64e, State: Initialized, Role: FOLLOWER
I20260810 23:41:47.237365 23555 consensus_queue.cc:260] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.246101 23553 raft_consensus.cc:348] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.246763 23553 raft_consensus.cc:374] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:47.246868 23553 raft_consensus.cc:729] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9e9ee37f1344b0976e7a34c1d0579b, State: Initialized, Role: FOLLOWER
I20260810 23:41:47.247641 23553 consensus_queue.cc:260] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.250909 23553 ts_tablet_manager.cc:1428] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Time spent starting tablet: real 0.020s user 0.008s sys 0.008s
I20260810 23:41:47.262241 23555 ts_tablet_manager.cc:1428] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Time spent starting tablet: real 0.037s user 0.008s sys 0.020s
I20260810 23:41:47.265717 23504 heartbeater.cc:503] Master 127.30.33.62:32879 was elected leader, sending a full tablet report...
I20260810 23:41:47.370297 23556 raft_consensus.cc:482] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:41:47.370731 23556 raft_consensus.cc:504] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.373076 23556 leader_election.cc:288] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb9e9ee37f1344b0976e7a34c1d0579b (127.30.33.2:37755), 2926dc869d0c415cb59c20d54553f64e (127.30.33.3:41401)
I20260810 23:41:47.384356 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.373770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:36545 (local address 127.30.33.3:41401)
0810 23:41:47.373970 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.373976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.374543 (+ 567us) server_negotiation.cc:407] Connection header received
0810 23:41:47.374878 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.374884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.374968 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.375034 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:41:47.375972 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.377120 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.381861 (+ 4741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.383072 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.383182 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.383298 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.383939 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.384092 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:41:47.385349 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.373571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:41401 (local address 127.30.33.1:36545)
0810 23:41:47.374429 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:47.374449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:47.374673 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:47.375260 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:47.375266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:47.375296 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:41:47.375785 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.375799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.377322 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.377328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.381461 (+ 4133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.381475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.383533 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.383542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.383738 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.383773 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:41:47.384867 (+ 1094us) client_negotiation.cc:240] Negotiation successful
0810 23:41:47.385017 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":1263,"threads_started":1}
I20260810 23:41:47.386696 23418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7f8ca2093845c5960b7809e0f58445" candidate_uuid: "1f21ced16c10489cbae6a4a8192c1763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2926dc869d0c415cb59c20d54553f64e" is_pre_election: true
I20260810 23:41:47.387080 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.374506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:33387 (local address 127.30.33.2:37755)
0810 23:41:47.374698 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.374703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.380046 (+ 5343us) server_negotiation.cc:407] Connection header received
0810 23:41:47.380357 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.380363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.380457 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.380524 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:41:47.381267 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.382294 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.384793 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.385936 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.386042 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.386131 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.386703 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.386840 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:41:47.387349 23418 raft_consensus.cc:2454] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f21ced16c10489cbae6a4a8192c1763 in term 0.
I20260810 23:41:47.387805 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.374028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:37755 (local address 127.30.33.1:33387)
0810 23:41:47.379917 (+ 5889us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:47.379941 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:47.380170 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:47.380754 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:47.380759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:47.380787 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:41:47.381095 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.381104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.382463 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.382468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.384613 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:47.384629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.386339 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.386345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:47.386511 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.386541 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:41:47.387328 (+ 787us) client_negotiation.cc:240] Negotiation successful
0810 23:41:47.387495 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5766,"thread_start_us":5602,"threads_started":1}
I20260810 23:41:47.388577 22753 leader_election.cc:302] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f21ced16c10489cbae6a4a8192c1763, 2926dc869d0c415cb59c20d54553f64e; no voters:
I20260810 23:41:47.388808 23147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7f8ca2093845c5960b7809e0f58445" candidate_uuid: "1f21ced16c10489cbae6a4a8192c1763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" is_pre_election: true
I20260810 23:41:47.389071 23556 raft_consensus.cc:2781] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:41:47.389231 23556 raft_consensus.cc:482] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:41:47.389333 23556 raft_consensus.cc:3037] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:47.389417 23147 raft_consensus.cc:2454] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f21ced16c10489cbae6a4a8192c1763 in term 0.
I20260810 23:41:47.392001 23556 raft_consensus.cc:504] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.392570 23556 leader_election.cc:288] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [CANDIDATE]: Term 1 election: Requested vote from peers bb9e9ee37f1344b0976e7a34c1d0579b (127.30.33.2:37755), 2926dc869d0c415cb59c20d54553f64e (127.30.33.3:41401)
I20260810 23:41:47.393019 23147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7f8ca2093845c5960b7809e0f58445" candidate_uuid: "1f21ced16c10489cbae6a4a8192c1763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b"
I20260810 23:41:47.393249 23147 raft_consensus.cc:3037] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:47.393491 23418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7f8ca2093845c5960b7809e0f58445" candidate_uuid: "1f21ced16c10489cbae6a4a8192c1763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2926dc869d0c415cb59c20d54553f64e"
I20260810 23:41:47.393703 23418 raft_consensus.cc:3037] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:47.396598 23147 raft_consensus.cc:2454] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f21ced16c10489cbae6a4a8192c1763 in term 1.
I20260810 23:41:47.397217 22758 leader_election.cc:302] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f21ced16c10489cbae6a4a8192c1763, bb9e9ee37f1344b0976e7a34c1d0579b; no voters:
I20260810 23:41:47.397302 23418 raft_consensus.cc:2454] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f21ced16c10489cbae6a4a8192c1763 in term 1.
I20260810 23:41:47.397959 23556 raft_consensus.cc:2781] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:41:47.398448 23556 raft_consensus.cc:686] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 1 LEADER]: Becoming Leader. State: Replica: 1f21ced16c10489cbae6a4a8192c1763, State: Running, Role: LEADER
I20260810 23:41:47.399145 23556 consensus_queue.cc:237] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } }
I20260810 23:41:47.412865 22600 catalog_manager.cc:5495] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f21ced16c10489cbae6a4a8192c1763 (127.30.33.1). New cstate: current_term: 1 leader_uuid: "1f21ced16c10489cbae6a4a8192c1763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f21ced16c10489cbae6a4a8192c1763" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 } health_report { overall_health: UNKNOWN } } }
W20260810 23:41:47.417454 23505 tablet.cc:2369] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:41:47.418828 22961 tablet.cc:2369] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:41:47.436308 23233 tablet.cc:2369] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:41:47.491559 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.484896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.30.33.1:34769)
0810 23:41:47.485171 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.485177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.485199 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:41:47.485376 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.485381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.485496 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.485648 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:41:47.486001 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.487045 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.488524 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.489162 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.489303 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.490019 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:41:47.490813 (+ 794us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:41:47.490969 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.491125 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.491233 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260810 23:41:47.500787 23418 raft_consensus.cc:1261] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 1 FOLLOWER]: Refusing update from remote peer 1f21ced16c10489cbae6a4a8192c1763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:41:47.500869 23147 raft_consensus.cc:1261] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 1 FOLLOWER]: Refusing update from remote peer 1f21ced16c10489cbae6a4a8192c1763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:41:47.501855 23556 consensus_queue.cc:1035] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9e9ee37f1344b0976e7a34c1d0579b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:41:47.502200 23577 consensus_queue.cc:1035] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2926dc869d0c415cb59c20d54553f64e" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:41:47.519904 23582 mvcc.cc:205] Tried to move back new op lower bound from 7317116139511439360 to 7317116139145834496. Current Snapshot: MvccSnapshot[applied={T|T < 7317116139511439360}]
I20260810 23:41:47.526602 23584 mvcc.cc:205] Tried to move back new op lower bound from 7317116139511439360 to 7317116139145834496. Current Snapshot: MvccSnapshot[applied={T|T < 7317116139511439360}]
I20260810 23:41:47.541721 22594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:41:47.542030 22594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:41:47.546411 22594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac7ad679dd274b2ba16451ecc982317b: Sending DeleteTablet for 3 replicas of tablet dc7f8ca2093845c5960b7809e0f58445
I20260810 23:41:47.547765 22847 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7f8ca2093845c5960b7809e0f58445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:41:47 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:60380
I20260810 23:41:47.548544 23110 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7f8ca2093845c5960b7809e0f58445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:41:47 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:34764
I20260810 23:41:47.548552 23374 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7f8ca2093845c5960b7809e0f58445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:41:47 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:52774
I20260810 23:41:47.550645 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 22662
I20260810 23:41:47.564107 23599 tablet_replica.cc:330] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: stopping tablet replica
I20260810 23:41:47.564790 23599 raft_consensus.cc:2229] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:41:47.566097 23598 tablet_replica.cc:330] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: stopping tablet replica
I20260810 23:41:47.566869 23598 raft_consensus.cc:2229] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:41:47.567850 23598 raft_consensus.cc:2258] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:41:47.569434 23599 raft_consensus.cc:2258] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:41:47.570469 23598 ts_tablet_manager.cc:1905] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:41:47.577301 23599 ts_tablet_manager.cc:1905] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:41:47.578015 23598 ts_tablet_manager.cc:1918] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:41:47.578291 23598 log.cc:1196] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/dc7f8ca2093845c5960b7809e0f58445
I20260810 23:41:47.578837 23598 ts_tablet_manager.cc:1939] T dc7f8ca2093845c5960b7809e0f58445 P 1f21ced16c10489cbae6a4a8192c1763: Deleting consensus metadata
I20260810 23:41:47.580869 22573 catalog_manager.cc:4841] TS 1f21ced16c10489cbae6a4a8192c1763 (127.30.33.1:34769): tablet dc7f8ca2093845c5960b7809e0f58445 (table test-table [id=178716e7121149ae9835c86c501a070c]) successfully deleted
I20260810 23:41:47.583261 23604 tablet_replica.cc:330] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: stopping tablet replica
I20260810 23:41:47.583850 23604 raft_consensus.cc:2229] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:41:47.584468 23604 raft_consensus.cc:2258] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:41:47.585824 23599 ts_tablet_manager.cc:1918] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:41:47.586794 23604 ts_tablet_manager.cc:1905] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:41:47.592151 23599 log.cc:1196] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/dc7f8ca2093845c5960b7809e0f58445
I20260810 23:41:47.592933 23599 ts_tablet_manager.cc:1939] T dc7f8ca2093845c5960b7809e0f58445 P bb9e9ee37f1344b0976e7a34c1d0579b: Deleting consensus metadata
I20260810 23:41:47.595261 22569 catalog_manager.cc:4841] TS bb9e9ee37f1344b0976e7a34c1d0579b (127.30.33.2:37755): tablet dc7f8ca2093845c5960b7809e0f58445 (table test-table [id=178716e7121149ae9835c86c501a070c]) successfully deleted
I20260810 23:41:47.599421 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.552233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.30.33.1:34769)
0810 23:41:47.552430 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.552437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.576295 (+ 23858us) server_negotiation.cc:407] Connection header received
0810 23:41:47.576871 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.576877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.576972 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.577052 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:47.579998 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.581165 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.583112 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.583853 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.583985 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.589539 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:47.589560 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:47.589580 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:47.589637 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:47.593899 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:47.596613 (+ 2714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:47.596625 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:47.596628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:47.596881 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:47.597550 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:47.597557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:47.597560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:47.598049 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:47.598371 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.598758 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.599025 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:41:47.600147 22950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:47.602087 23604 ts_tablet_manager.cc:1918] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:41:47.602339 23604 log.cc:1196] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/dc7f8ca2093845c5960b7809e0f58445
I20260810 23:41:47.602977 23604 ts_tablet_manager.cc:1939] T dc7f8ca2093845c5960b7809e0f58445 P 2926dc869d0c415cb59c20d54553f64e: Deleting consensus metadata
I20260810 23:41:47.605330 22566 catalog_manager.cc:4841] TS 2926dc869d0c415cb59c20d54553f64e (127.30.33.3:41401): tablet dc7f8ca2093845c5960b7809e0f58445 (table test-table [id=178716e7121149ae9835c86c501a070c]) successfully deleted
I20260810 23:41:47.709300 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 22662
I20260810 23:41:47.737077 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 22965
I20260810 23:41:47.751912 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.737973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.30.33.2:37755)
0810 23:41:47.738196 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.738202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.738223 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:41:47.738323 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.738329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.738400 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.738482 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:47.740011 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.740778 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.742184 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.742595 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.742714 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.745690 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:47.745708 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:47.745726 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:47.745768 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:47.748706 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:47.749522 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:47.749531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:47.749533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:47.749686 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:47.750079 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:47.750083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:47.750085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:47.750504 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:47.750792 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.751150 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.751385 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:41:47.752714 23221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:47.842429 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 22965
I20260810 23:41:47.873884 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 23240
I20260810 23:41:47.888659 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:47.874723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.30.33.3:41401)
0810 23:41:47.874906 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:47.874911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:47.874931 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:41:47.875047 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:47.875051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:47.875136 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:47.875229 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:47.876672 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.877423 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.878802 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:47.879245 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:47.879361 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:47.882199 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:47.882212 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:47.882229 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:47.882271 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:47.885350 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:47.886094 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:47.886102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:47.886105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:47.886257 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:47.886664 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:47.886668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:47.886670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:47.887115 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:47.887398 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:47.887833 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 23:41:47.888088 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:41:47.889369 23497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:47.979696 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 23240
I20260810 23:41:48.004623 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 22533
I20260810 23:41:48.005995 22644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:48.119153 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 22533
I20260810 23:41:48.152977 30852 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:41:48.488884 30852 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:41:48.489078 30852 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:41:48.845263 30852 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:41:48.845 UTC [20761] LOG: received smart shutdown request
2026-08-10 23:41:48.848 UTC [20761] LOG: background worker "logical replication launcher" (PID 20777) exited with exit code 1
2026-08-10 23:41:48.848 UTC [20764] LOG: shutting down
2026-08-10 23:41:49.003 UTC [20761] LOG: database system is shut down
2026-08-10T23:41:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56086 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:41:49 dist-test-slave-pww0 krb5kdc[23624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:41:49 dist-test-slave-pww0 krb5kdc[23624](info): set up 2 sockets
Aug 10 23:41:49 dist-test-slave-pww0 krb5kdc[23624](info): commencing operation
krb5kdc: starting...
W20260810 23:41:50.423137 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:41:50 dist-test-slave-pww0 krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:41:50.723711 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37303
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:37303,127.30.33.61:39913,127.30.33.60:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:51.087440 23654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:51.087819 23654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:51.087970 23654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:51.095599 23654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:41:51.095741 23654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:51.095824 23654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:51.095903 23654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:41:51.095968 23654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:41:51.107100 23654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:37303,127.30.33.61:39913,127.30.33.60:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:51.108719 23654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:51.110181 23654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:51.123129 23670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:51.123179 23664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:51.123206 23667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:51.124185 23654 server_base.cc:1034] running on GCE node
I20260810 23:41:51.125051 23654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:51.128636 23654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:51.129942 23654 hybrid_clock.cc:648] HybridClock initialized: now 1786405311129868 us; error 70 us; skew 500 ppm
I20260810 23:41:51.130445 23654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:51 dist-test-slave-pww0 krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:51.139997 23654 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:41:51.141654 23654 webserver.cc:460] Webserver started at http://127.30.33.62:40959/ using document root <none> and password file <none>
I20260810 23:41:51.142408 23654 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:51.142518 23654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:51.142925 23654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:41:51.146539 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "bf52c4e1d48748338c12a318d4bd11b8"
format_stamp: "Formatted at 2026-08-10 23:41:51 on dist-test-slave-pww0"
server_key: "d44016b715ddb73535dfecc4eafefee7"
server_key_iv: "f52f4816b7f235c0b484ec635adf6398"
server_key_version: "encryptionkey@0"
I20260810 23:41:51.147447 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "bf52c4e1d48748338c12a318d4bd11b8"
format_stamp: "Formatted at 2026-08-10 23:41:51 on dist-test-slave-pww0"
server_key: "d44016b715ddb73535dfecc4eafefee7"
server_key_iv: "f52f4816b7f235c0b484ec635adf6398"
server_key_version: "encryptionkey@0"
I20260810 23:41:51.153333 23654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:41:51.157330 23680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:51.158879 23654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:41:51.159065 23654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "bf52c4e1d48748338c12a318d4bd11b8"
format_stamp: "Formatted at 2026-08-10 23:41:51 on dist-test-slave-pww0"
server_key: "d44016b715ddb73535dfecc4eafefee7"
server_key_iv: "f52f4816b7f235c0b484ec635adf6398"
server_key_version: "encryptionkey@0"
I20260810 23:41:51.159279 23654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:51.206184 23654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:51.210955 23654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:51.211246 23654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:51.230412 23654 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:37303
I20260810 23:41:51.230445 23766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:37303 every 8 connection(s)
I20260810 23:41:51.231873 23654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:41:51.234946 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 23654
I20260810 23:41:51.235211 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:41:51.236070 30852 external_mini_cluster.cc:1428] Setting key fe6a3c9d3ff79d1f1ff5c6eec0d4d4cd
I20260810 23:41:51.236364 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:51.245735 23767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:41:51 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.61@KRBTEST.COM" created.
I20260810 23:41:51.271390 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.249783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.30.33.62:37303)
0810 23:41:51.251650 (+ 1867us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:51.251658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:51.251717 (+ 59us) server_negotiation.cc:407] Connection header received
0810 23:41:51.253362 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:51.253371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:51.253744 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:51.253975 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:51.255190 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.257108 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.258855 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.259793 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.259974 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.263450 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:51.263486 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:51.263505 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:51.263547 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:51.266892 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.267679 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.267687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.267689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.267881 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.268331 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.268336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.268339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.269030 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:51.269191 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:51.269849 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 23:41:51.270183 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":1630,"spinlock_wait_cycles":13696,"thread_start_us":1400,"threads_started":1}
I20260810 23:41:51.271413 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.249784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37303 (local address 127.0.0.1:33836)
0810 23:41:51.250854 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.250919 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:51.252939 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:51.253992 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:51.253997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:51.254600 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:51.254982 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.255000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.257268 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.257272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.258696 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.258711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.260006 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.260011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.260133 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.260795 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:51.260819 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:51.263135 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:51.267045 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.267058 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.267070 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.267550 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.268010 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.268013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.268016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.268208 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.269198 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:51.269207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:51.269462 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 23:41:51.269760 (+ 298us) client_negotiation.cc:240] Negotiation successful
0810 23:41:51.270370 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":90,"thread_start_us":453,"threads_started":1}
I20260810 23:41:51.274655 23767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:41:51.275918 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.275359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46824)
0810 23:41:51.275612 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.275766 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 23:41:51.276176 23681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
I20260810 23:41:51.277860 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.277423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46830)
0810 23:41:51.277612 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.277698 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:41:51.278061 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:39913: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab.
Entry for principal kudu/127.30.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab.
I20260810 23:41:51.303866 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:39913
--webserver_interface=127.30.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab
--principal=kudu/127.30.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:37303,127.30.33.61:39913,127.30.33.60:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:41:51.327576 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } attempt: 1
I20260810 23:41:51.328981 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.328384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46838)
0810 23:41:51.328625 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.328767 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:41:51.330287 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.329797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46844)
0810 23:41:51.329958 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.330056 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:41:51.330525 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:39913: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
I20260810 23:41:51.398852 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } attempt: 2
I20260810 23:41:51.400238 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.399686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46850)
0810 23:41:51.399937 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.400051 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 23:41:51.401480 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.401042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46858)
0810 23:41:51.401199 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.401309 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:41:51.401751 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:39913: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
I20260810 23:41:51.493039 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } attempt: 3
I20260810 23:41:51.494307 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.493731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46862)
0810 23:41:51.493985 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.494124 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 23:41:51.495517 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.495036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46864)
0810 23:41:51.495195 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.495290 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:41:51.495761 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:39913: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
I20260810 23:41:51.639034 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } attempt: 4
I20260810 23:41:51.640406 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.639760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46868)
0810 23:41:51.640112 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.640236 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260810 23:41:51.641594 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.641152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:46880)
0810 23:41:51.641308 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.641429 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:41:51.641808 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:39913: Network error: Client connection negotiation failed: client connection to 127.30.33.61:39913: connect: Connection refused (error 111)
W20260810 23:41:51.650148 23779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:51.650484 23779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:51.650586 23779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:51.658216 23779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:41:51.658344 23779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:51.658402 23779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:51.658457 23779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:41:51.658501 23779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:41:51.669070 23779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:37303,127.30.33.61:39913,127.30.33.60:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab
--principal=kudu/127.30.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:51.670444 23779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:51.671787 23779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:51.684332 23788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:51.684268 23790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:51.684753 23792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:51.685554 23779 server_base.cc:1034] running on GCE node
I20260810 23:41:51.686364 23779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:51.687855 23779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:51.689064 23779 hybrid_clock.cc:648] HybridClock initialized: now 1786405311689023 us; error 33 us; skew 500 ppm
I20260810 23:41:51.689411 23779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:51 dist-test-slave-pww0 krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:51.692619 23779 init.cc:377] Logged in from keytab as kudu/127.30.33.61@KRBTEST.COM (short username kudu)
I20260810 23:41:51.694165 23779 webserver.cc:460] Webserver started at http://127.30.33.61:45193/ using document root <none> and password file <none>
I20260810 23:41:51.694911 23779 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:51.695029 23779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:51.695339 23779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:41:51.698716 23779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data/instance:
uuid: "a21e3d3bb17444048716f5d63eb341f4"
format_stamp: "Formatted at 2026-08-10 23:41:51 on dist-test-slave-pww0"
server_key: "22a9322b1a34f6b82adb361efb3605ab"
server_key_iv: "64af9ab79504ae88007c8a19c16ca1ac"
server_key_version: "encryptionkey@0"
I20260810 23:41:51.699623 23779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/wal/instance:
uuid: "a21e3d3bb17444048716f5d63eb341f4"
format_stamp: "Formatted at 2026-08-10 23:41:51 on dist-test-slave-pww0"
server_key: "22a9322b1a34f6b82adb361efb3605ab"
server_key_iv: "64af9ab79504ae88007c8a19c16ca1ac"
server_key_version: "encryptionkey@0"
I20260810 23:41:51.705235 23779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:41:51.708984 23799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:51.710433 23779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:41:51.710604 23779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/wal
uuid: "a21e3d3bb17444048716f5d63eb341f4"
format_stamp: "Formatted at 2026-08-10 23:41:51 on dist-test-slave-pww0"
server_key: "22a9322b1a34f6b82adb361efb3605ab"
server_key_iv: "64af9ab79504ae88007c8a19c16ca1ac"
server_key_version: "encryptionkey@0"
I20260810 23:41:51.710819 23779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:51.725955 23779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:51.730026 23779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:51.730309 23779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:51.748533 23779 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.61:39913
I20260810 23:41:51.748556 23897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.61:39913 every 8 connection(s)
I20260810 23:41:51.749992 23779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
I20260810 23:41:51.752969 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 23779
I20260810 23:41:51.753186 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-1/wal/instance
I20260810 23:41:51.753793 30852 external_mini_cluster.cc:1428] Setting key 08831801301edc9200f11c34d11c2f81
I20260810 23:41:51.754575 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:51.764072 23898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:41:51 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.60@KRBTEST.COM" created.
I20260810 23:41:51.786808 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.767731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.30.33.62:37303)
0810 23:41:51.767917 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:51.767922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:51.769503 (+ 1581us) server_negotiation.cc:407] Connection header received
0810 23:41:51.771692 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:51.771699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:51.771797 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:51.771890 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:51.773691 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.774387 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.776117 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.776553 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.776677 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.780758 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:51.780763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:51.780766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:51.780805 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:51.783391 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.784233 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.784238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.784240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.784373 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.784843 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.784847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.784849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.785237 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:51.785413 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:51.786228 (+ 815us) server_negotiation.cc:299] Negotiation successful
0810 23:41:51.786476 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:41:51.787783 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.768404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37303 (local address 127.0.0.1:48266)
0810 23:41:51.769302 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.769355 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:51.771307 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:51.772185 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:51.772196 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:51.772809 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:51.773477 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.773498 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.774565 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.774569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.775944 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.775960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.776833 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.776841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.777060 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.777836 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:51.777860 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:51.780488 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:51.783566 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.783577 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.783595 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.784076 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.784488 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.784493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.784496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.784711 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.785419 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:51.785426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:51.785810 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 23:41:51.786069 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 23:41:51.786514 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":377,"threads_started":1}
I20260810 23:41:51.789705 23898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:41:51 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:41:51.805619 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.790445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:40104)
0810 23:41:51.791395 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.791412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:51.791547 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:51.792069 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:51.792074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:51.792464 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:51.792621 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.792630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.794384 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.794388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.795269 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.795283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.796404 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.796409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.796603 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.797229 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:51.797248 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:51.799631 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:51.802852 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.802857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.802861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.803351 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.803757 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.803763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.803767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.803968 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.804616 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:51.804621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:51.804769 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 23:41:51.804905 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:41:51.805222 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840}
I20260810 23:41:51.805648 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.790746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.30.33.61:39913)
0810 23:41:51.791525 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:51.791533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:51.791568 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:41:51.791640 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:51.791647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:51.791890 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:51.792045 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:51.792771 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.794212 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.795415 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.796238 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.796362 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.799853 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:51.799866 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:51.799881 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:51.799912 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:51.802692 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.803509 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.803519 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.803523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.803648 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.804080 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.804083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.804085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.804464 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:51.804631 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:51.805063 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 23:41:51.805346 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":563,"thread_start_us":395,"threads_started":1}
I20260810 23:41:51.808245 23898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:41:51.809630 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.809079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50104)
0810 23:41:51.809276 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.809443 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 23:41:51.809963 23802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:51.811705 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.811311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50112)
0810 23:41:51.811514 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.811597 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:41:51.811936 23898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab.
Entry for principal kudu/127.30.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab.
I20260810 23:41:51.822479 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:45959
--webserver_interface=127.30.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab
--principal=kudu/127.30.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:37303,127.30.33.61:39913,127.30.33.60:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:41:51.861253 23898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 1
I20260810 23:41:51.862380 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.861902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50116)
0810 23:41:51.862100 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.862209 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:41:51.863489 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.863086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50120)
0810 23:41:51.863257 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.863342 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:41:51.863698 23898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:51.931931 23898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 2
I20260810 23:41:51.933028 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.932582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50124)
0810 23:41:51.932776 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.932871 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:41:51.934053 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.933704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50130)
0810 23:41:51.933864 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.933932 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 23:41:51.934265 23898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:51.941069 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } attempt: 5
Aug 10 23:41:51 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:41:51.956686 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.941755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:40110)
0810 23:41:51.942012 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.942037 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:51.942202 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:51.942550 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:51.942556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:51.943090 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:51.943336 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.943349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.944409 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.944414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.945817 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:51.945835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.946591 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.946597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:51.946808 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.947523 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:51.947549 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:51.950109 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:51.954006 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.954011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.954014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.954463 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.954902 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:51.954905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:51.954908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:51.955045 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:51.955750 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:51.955755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:51.955973 (+ 218us) client_negotiation.cc:769] Sending connection context
0810 23:41:51.956150 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 23:41:51.956426 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:41:51.956858 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.942015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.30.33.61:39913)
0810 23:41:51.942182 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:51.942186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:51.942206 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:41:51.942326 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:51.942330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:51.942393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:51.942476 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:51.943563 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.944265 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.945966 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:51.946432 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:51.946530 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:51.950379 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:51.950387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:51.950392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:51.950439 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:51.953780 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.954607 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.954613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.954617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.954795 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:51.955144 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:51.955149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:51.955152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:51.955584 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:51.955743 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:51.956294 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 23:41:51.956556 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:41:51.957988 23767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:41:51.959170 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.958664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50134)
0810 23:41:51.958864 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.958969 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:41:51.960255 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:51.959793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50146)
0810 23:41:51.959973 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:51.960106 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 23:41:51.960493 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:52.011751 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 1
I20260810 23:41:52.013092 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.012458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50158)
0810 23:41:52.012733 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.012871 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 23:41:52.014328 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.013888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50172)
0810 23:41:52.014055 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.014156 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:41:52.014612 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:52.025483 23898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 3
I20260810 23:41:52.026628 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.026130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50176)
0810 23:41:52.026372 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.026469 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:41:52.027925 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.027418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50180)
0810 23:41:52.027713 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.027804 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
W20260810 23:41:52.028173 23898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:52.082855 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 2
I20260810 23:41:52.084190 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.083663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50186)
0810 23:41:52.083902 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.083993 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:41:52.085237 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.084883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50202)
0810 23:41:52.085049 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.085123 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:41:52.085453 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:52.171396 23898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 4
I20260810 23:41:52.172690 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.172163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50212)
0810 23:41:52.172418 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.172514 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:41:52.174062 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.173663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50214)
0810 23:41:52.173855 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.173938 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 23:41:52.174337 23898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
W20260810 23:41:52.182749 23906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:52.183079 23906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:52.183228 23906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:52.190951 23906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:41:52.191093 23906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:52.191174 23906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:52.191248 23906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:41:52.191311 23906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:41:52.191651 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 3
I20260810 23:41:52.192764 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.192273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50220)
0810 23:41:52.192478 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.192574 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:41:52.193909 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.193493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50222)
0810 23:41:52.193710 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.193792 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260810 23:41:52.194147 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:45959: Network error: Client connection negotiation failed: client connection to 127.30.33.60:45959: connect: Connection refused (error 111)
I20260810 23:41:52.202240 23906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:37303,127.30.33.61:39913,127.30.33.60:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab
--principal=kudu/127.30.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:52.203706 23906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:52.205102 23906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:52.217265 23924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:52.217178 23925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:52.217296 23931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:52.218282 23906 server_base.cc:1034] running on GCE node
I20260810 23:41:52.219189 23906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:52.220970 23906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:52.222193 23906 hybrid_clock.cc:648] HybridClock initialized: now 1786405312222132 us; error 58 us; skew 500 ppm
I20260810 23:41:52.222585 23906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405312, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:52.225890 23906 init.cc:377] Logged in from keytab as kudu/127.30.33.60@KRBTEST.COM (short username kudu)
I20260810 23:41:52.227497 23906 webserver.cc:460] Webserver started at http://127.30.33.60:33455/ using document root <none> and password file <none>
I20260810 23:41:52.228209 23906 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:52.228325 23906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:52.228590 23906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:41:52.232489 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data/instance:
uuid: "279d7f3191084d3293cf0a89c1914c20"
format_stamp: "Formatted at 2026-08-10 23:41:52 on dist-test-slave-pww0"
server_key: "0cbe54d129eda26d19957f99e391d806"
server_key_iv: "68de6334df4ae648e2b6012237094d98"
server_key_version: "encryptionkey@0"
I20260810 23:41:52.233551 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/wal/instance:
uuid: "279d7f3191084d3293cf0a89c1914c20"
format_stamp: "Formatted at 2026-08-10 23:41:52 on dist-test-slave-pww0"
server_key: "0cbe54d129eda26d19957f99e391d806"
server_key_iv: "68de6334df4ae648e2b6012237094d98"
server_key_version: "encryptionkey@0"
I20260810 23:41:52.240890 23906 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260810 23:41:52.244827 23949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:52.246551 23906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:41:52.246747 23906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/wal
uuid: "279d7f3191084d3293cf0a89c1914c20"
format_stamp: "Formatted at 2026-08-10 23:41:52 on dist-test-slave-pww0"
server_key: "0cbe54d129eda26d19957f99e391d806"
server_key_iv: "68de6334df4ae648e2b6012237094d98"
server_key_version: "encryptionkey@0"
I20260810 23:41:52.246954 23906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:52.262316 23906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:52.267146 23906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:52.267491 23906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:52.285971 23906 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.60:45959
I20260810 23:41:52.286005 24046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.60:45959 every 8 connection(s)
I20260810 23:41:52.287421 23906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
I20260810 23:41:52.291282 24048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:52.292713 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 23906
I20260810 23:41:52.292922 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786405091056293-30852-0/minicluster-data/master-2/wal/instance
I20260810 23:41:52.293473 30852 external_mini_cluster.cc:1428] Setting key 26947efb03c7884733bf55b3c9bbf22c
I20260810 23:41:52.301604 24048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405312, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:52.318662 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.295554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.30.33.60:45959)
0810 23:41:52.296490 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.296512 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.296567 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:41:52.298373 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.298402 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.298910 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.299327 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.301451 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.302923 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.304640 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.305793 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.305977 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.310735 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.310747 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.310763 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.310816 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.314114 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.315029 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.315041 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.315051 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.315291 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.315767 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.315773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.315777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.316376 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.316671 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.317142 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.317571 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":435,"threads_started":1}
I20260810 23:41:52.320925 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.304535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.30.33.62:37303)
0810 23:41:52.305892 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.305898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.305916 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:41:52.305975 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.305980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.306067 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.306180 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.307716 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.308568 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.310205 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.310772 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.310906 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.314605 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.314612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.314615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.314699 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.317917 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.318700 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.318707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.318711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.318920 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.319387 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.319393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.319396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.319768 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.319916 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.320432 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.320612 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":1104,"threads_started":1}
I20260810 23:41:52.321187 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.304438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:37303 (local address 127.0.0.1:48282)
0810 23:41:52.305215 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.305276 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:52.305450 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:52.306281 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:52.306288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:52.307137 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:52.307554 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.307570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.308738 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.308743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.310048 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.310065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.310978 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.310983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.311125 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.311920 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:52.311942 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:52.314346 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:52.318056 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.318070 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.318079 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.318546 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.319086 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.319093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.319096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.319258 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.319915 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:52.319921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:52.320108 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 23:41:52.320337 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 23:41:52.320695 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":324,"threads_started":1}
I20260810 23:41:52.322654 24048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405312, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:52.336786 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.323323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:39913 (local address 127.0.0.1:40114)
0810 23:41:52.323531 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.323550 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:52.323645 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:52.323941 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:52.323945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:52.324392 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:52.324589 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.324600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.325666 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.325671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.326599 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.326612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.327612 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.327619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.327847 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.328482 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:52.328499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:52.330462 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:52.334026 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.334032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.334036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.334536 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.334985 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.334989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.334993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.335179 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.335844 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:52.335849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:52.336023 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 23:41:52.336183 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 23:41:52.336422 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 23:41:52.336902 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.323478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.30.33.61:39913)
0810 23:41:52.323645 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.323649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.323667 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:41:52.323747 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.323751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.323812 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.323896 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.324758 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.325513 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.326773 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.327410 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.327592 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.330691 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.330699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.330705 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.330751 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.333874 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.334689 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.334694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.334697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.334866 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.335289 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.335295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.335298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.335725 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.335912 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.336326 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.336599 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:41:52.338090 24048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:41:52.338400 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.322554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.30.33.62:37303)
0810 23:41:52.322721 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.322726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.322871 (+ 145us) server_negotiation.cc:407] Connection header received
0810 23:41:52.323038 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.323043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.323119 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.323208 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.324922 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.325675 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.327162 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.327698 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.327823 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.332192 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.332197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.332200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.332242 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.334989 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.335910 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.335916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.335919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.336067 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.336562 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.336568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.336571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.337085 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.337350 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.337859 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.338105 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405312, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:41:52.354549 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.339102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.30.33.60:45959)
0810 23:41:52.339255 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.339261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.339333 (+ 72us) server_negotiation.cc:407] Connection header received
0810 23:41:52.339521 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.339526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.339590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.339669 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.340566 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.341621 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.343138 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.343716 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.343843 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.347402 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.347410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.347415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.347484 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.350841 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.352083 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.352089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.352093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.352253 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.352780 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.352785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.352787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.353227 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.353360 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.353915 (+ 555us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.354291 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":69}
I20260810 23:41:52.354591 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.339072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50236)
0810 23:41:52.339254 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.339272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:52.339390 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:52.339698 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:52.339703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:52.340191 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:52.340378 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.340390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.341770 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.341774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.342964 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.342979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.343879 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.343884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.344021 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.344625 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:52.344644 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:52.347191 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:52.351364 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.351370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.351374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.351941 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.352415 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.352419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.352423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.352597 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.353401 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:52.353407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:52.353611 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 23:41:52.353785 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 23:41:52.354136 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:41:52.358016 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.340089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.30.33.61:39913)
0810 23:41:52.340259 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.340264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.340282 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:41:52.340356 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.340361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.340435 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.340528 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.342222 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.343286 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.344810 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.345450 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.345588 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.351111 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.351117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.351126 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.351168 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.354930 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.355797 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.355802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.355805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.355967 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.356429 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.356435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.356438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.356849 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.357166 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.357408 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.357662 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:41:52.360710 24048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: Bootstrap starting.
I20260810 23:41:52.363420 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:41:52.364790 24048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:52.366184 24048 log.cc:824] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:52.369416 24048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: No bootstrap required, opened a new log
I20260810 23:41:52.371338 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 4
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
I20260810 23:41:52.390892 24048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.391600 24048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:52.391820 24048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279d7f3191084d3293cf0a89c1914c20, State: Initialized, Role: FOLLOWER
I20260810 23:41:52.392063 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.371957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50254)
0810 23:41:52.372234 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.372265 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:52.372386 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:52.372915 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:52.372921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:52.373428 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:52.374935 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.374948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.376645 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.376652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.378254 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.378272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.379708 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.379715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.379937 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.380715 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:52.380781 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:52.384270 (+ 3489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:52.388772 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.388778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.388781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.389394 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.389896 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.389901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.389905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.390085 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.390907 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:52.390913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:52.391096 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 23:41:52.391245 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 23:41:52.391602 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1245}
I20260810 23:41:52.392784 24048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.393136 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.372029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.30.33.60:45959)
0810 23:41:52.372224 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.372230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.372322 (+ 92us) server_negotiation.cc:407] Connection header received
0810 23:41:52.372516 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.372522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.372597 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.372719 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.375140 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.376416 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.378476 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.379257 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.379381 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.384584 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.384594 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.384600 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.384653 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.388571 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.389553 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.389559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.389563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.389748 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.390222 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.390227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.390230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.390652 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.390820 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.392522 (+ 1702us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.392792 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:41:52.400058 23767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: Bootstrap starting.
I20260810 23:41:52.404291 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.369299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.30.33.62:37303)
0810 23:41:52.369495 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.369501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.370091 (+ 590us) server_negotiation.cc:407] Connection header received
0810 23:41:52.370420 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.370427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.370521 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.370599 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.373295 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.374583 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.391233 (+ 16650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.391938 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.392059 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.396039 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.396047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.396051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.396102 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.400012 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.400993 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.400999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.401002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.401169 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.401692 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.401697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.401700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.402127 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.402423 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.403351 (+ 928us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.403617 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":70}
I20260810 23:41:52.404723 24048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:41:52.401638 24078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.405596 24078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:41:52.406118 23767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:52.407693 23767 log.cc:824] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:52.409809 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.370267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.30.33.61:39913)
0810 23:41:52.370451 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.370456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.388516 (+ 18060us) server_negotiation.cc:407] Connection header received
0810 23:41:52.388807 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.388813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.388964 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.389038 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.391791 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.392877 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.395233 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.396018 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.396129 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.402027 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.402035 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.402040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.402087 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.405602 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.406899 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.406908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.406912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.407111 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.407679 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.407685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.407689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.408110 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.408395 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.409094 (+ 699us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.409390 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:41:52.414466 23767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: No bootstrap required, opened a new log
I20260810 23:41:52.418174 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.375703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.30.33.60:45959)
0810 23:41:52.387341 (+ 11638us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.387348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.387377 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:41:52.387451 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.387511 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.387606 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.387752 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.397541 (+ 9789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.398895 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.401868 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.402531 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.402662 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.409662 (+ 7000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.409671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.409677 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.409729 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.413887 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.415155 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.415162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.415165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.415371 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.416045 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.416051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.416055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.416532 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.416815 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.417486 (+ 671us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.417745 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11450,"thread_start_us":12327,"threads_started":1}
I20260810 23:41:52.425382 23767 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
W20260810 23:41:52.425803 24095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:41:52.425971 24095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:41:52.425956 23767 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:52.426075 23767 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf52c4e1d48748338c12a318d4bd11b8, State: Initialized, Role: FOLLOWER
I20260810 23:41:52.426739 23767 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.427722 24101 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.428246 24101 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:41:52.429149 23767 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:41:52.421334 23906 master_runner.cc:186] Error getting master registration for 127.30.33.62:37303: OK, instance_id { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" instance_seqno: 1786405311224915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:41:52.431222 23906 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:41:52.449785 24117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:41:52.450104 24117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:41:52.441968 23654 master_runner.cc:186] Error getting master registration for 127.30.33.61:39913: OK, instance_id { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" instance_seqno: 1786405311743053 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:41:52.454017 23654 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:41:52.464407 23690 tablet.cc:2369] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:41:52.473574 23898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } attempt: 5
Aug 10 23:41:52 dist-test-slave-pww0 krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405311, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
I20260810 23:41:52.492681 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.474355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:45959 (local address 127.0.0.1:50268)
0810 23:41:52.474680 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:52.474702 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:52.474849 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:52.475275 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:52.475280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:52.475899 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:52.476125 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.476140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.477609 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.477615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.479103 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:52.479127 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.480307 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.480314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:52.480587 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.481322 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:52.481348 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:52.484867 (+ 3519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:52.489521 (+ 4654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.489527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.489531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.490174 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.490702 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:52.490708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:52.490712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:52.490904 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:52.491572 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:52.491578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:52.491752 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 23:41:52.491925 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 23:41:52.492266 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260810 23:41:52.492802 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:52.474473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.30.33.60:45959)
0810 23:41:52.474710 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:52.474716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:52.474793 (+ 77us) server_negotiation.cc:407] Connection header received
0810 23:41:52.475005 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:52.475011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:52.475101 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:52.475230 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:52.476317 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.477444 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.479341 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:52.480095 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:52.480236 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:52.485101 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:52.485109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:52.485114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:52.485158 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:52.489297 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.490367 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.490374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.490377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.490560 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:52.491030 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:52.491035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:52.491039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:52.491414 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:52.491616 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:52.492080 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 23:41:52.492347 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:41:52.499678 23898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4: Bootstrap starting.
I20260810 23:41:52.504213 23898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:52.505752 23898 log.cc:824] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:52.510773 23898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4: No bootstrap required, opened a new log
W20260810 23:41:52.519965 23959 tablet.cc:2369] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:41:52.522760 23898 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.523262 23898 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:52.523394 23898 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21e3d3bb17444048716f5d63eb341f4, State: Initialized, Role: FOLLOWER
I20260810 23:41:52.524238 23898 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.528795 24122 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.529366 24122 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:41:52.530642 23898 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:41:52.559386 23779 master_runner.cc:263] No leader master found from master a21e3d3bb17444048716f5d63eb341f4
I20260810 23:41:52.559604 23779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:41:52.560477 24152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:41:52.560631 24152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:41:52.694813 24122 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:41:52.695207 24122 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.697070 24122 leader_election.cc:288] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf52c4e1d48748338c12a318d4bd11b8 (127.30.33.62:37303), 279d7f3191084d3293cf0a89c1914c20 (127.30.33.60:45959)
I20260810 23:41:52.698566 23726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21e3d3bb17444048716f5d63eb341f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52c4e1d48748338c12a318d4bd11b8" is_pre_election: true
I20260810 23:41:52.699301 23726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a21e3d3bb17444048716f5d63eb341f4 in term 0.
I20260810 23:41:52.700412 23800 leader_election.cc:302] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a21e3d3bb17444048716f5d63eb341f4, bf52c4e1d48748338c12a318d4bd11b8; no voters:
I20260810 23:41:52.701310 24122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:41:52.701498 24122 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:41:52.701607 24122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:52.703118 24000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21e3d3bb17444048716f5d63eb341f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279d7f3191084d3293cf0a89c1914c20" is_pre_election: true
I20260810 23:41:52.703873 24000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a21e3d3bb17444048716f5d63eb341f4 in term 0.
I20260810 23:41:52.708313 24122 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.709954 23726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21e3d3bb17444048716f5d63eb341f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52c4e1d48748338c12a318d4bd11b8"
I20260810 23:41:52.710275 23726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:52.714323 23726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a21e3d3bb17444048716f5d63eb341f4 in term 1.
I20260810 23:41:52.715217 23800 leader_election.cc:302] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a21e3d3bb17444048716f5d63eb341f4, bf52c4e1d48748338c12a318d4bd11b8; no voters:
I20260810 23:41:52.722163 24000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a21e3d3bb17444048716f5d63eb341f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279d7f3191084d3293cf0a89c1914c20"
I20260810 23:41:52.722470 24000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:52.726425 24000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a21e3d3bb17444048716f5d63eb341f4 in term 1.
I20260810 23:41:52.728474 24122 leader_election.cc:288] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [CANDIDATE]: Term 1 election: Requested vote from peers bf52c4e1d48748338c12a318d4bd11b8 (127.30.33.62:37303), 279d7f3191084d3293cf0a89c1914c20 (127.30.33.60:45959)
I20260810 23:41:52.731664 24122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 23:41:52.733045 23811 tablet.cc:2369] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:41:52.734601 24122 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [term 1 LEADER]: Becoming Leader. State: Replica: a21e3d3bb17444048716f5d63eb341f4, State: Running, Role: LEADER
I20260810 23:41:52.735666 24122 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } }
I20260810 23:41:52.753448 24161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21e3d3bb17444048716f5d63eb341f4. Latest consensus state: current_term: 1 leader_uuid: "a21e3d3bb17444048716f5d63eb341f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.754042 24161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:52.774369 24164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:41:52.777318 24164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:41:52.803256 24000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [term 1 FOLLOWER]: Refusing update from remote peer a21e3d3bb17444048716f5d63eb341f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:41:52.806617 23726 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [term 1 FOLLOWER]: Refusing update from remote peer a21e3d3bb17444048716f5d63eb341f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:41:52.809923 24122 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:41:52.814134 24122 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:41:52.860752 24078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21e3d3bb17444048716f5d63eb341f4. Latest consensus state: current_term: 1 leader_uuid: "a21e3d3bb17444048716f5d63eb341f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.861114 24078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:41:52.863112 24165 mvcc.cc:205] Tried to move back new op lower bound from 7317116161227735040 to 7317116160979505152. Current Snapshot: MvccSnapshot[applied={T|T < 7317116161227735040}]
I20260810 23:41:52.866696 24169 mvcc.cc:205] Tried to move back new op lower bound from 7317116161227735040 to 7317116160979505152. Current Snapshot: MvccSnapshot[applied={T|T < 7317116161227735040}]
I20260810 23:41:52.872861 24101 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21e3d3bb17444048716f5d63eb341f4. Latest consensus state: current_term: 1 leader_uuid: "a21e3d3bb17444048716f5d63eb341f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.873289 24101 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:41:52.877637 24078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21e3d3bb17444048716f5d63eb341f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.878001 24078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:41:52.880913 24161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21e3d3bb17444048716f5d63eb341f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.881260 24161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:52.883653 24161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21e3d3bb17444048716f5d63eb341f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.883978 24161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:52.893151 24164 catalog_manager.cc:1347] Generated new cluster ID: 05c30684c8194129805b9367f5b8852e
I20260810 23:41:52.893307 24164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:41:52.903023 24101 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21e3d3bb17444048716f5d63eb341f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf52c4e1d48748338c12a318d4bd11b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37303 } } peers { permanent_uuid: "a21e3d3bb17444048716f5d63eb341f4" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 39913 } } peers { permanent_uuid: "279d7f3191084d3293cf0a89c1914c20" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 45959 } } }
I20260810 23:41:52.903307 24101 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:41:52.932601 24164 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:41:52.934584 24164 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:41:52.956836 24164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21e3d3bb17444048716f5d63eb341f4: Generated new TSK 0
I20260810 23:41:52.958278 24164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:41:53.428154 24095 catalog_manager.cc:1259] Loaded cluster ID: 05c30684c8194129805b9367f5b8852e
I20260810 23:41:53.428270 24095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: loading cluster ID for follower catalog manager: success
I20260810 23:41:53.431006 24095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: acquiring CA information for follower catalog manager: success
I20260810 23:41:53.433099 24095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 279d7f3191084d3293cf0a89c1914c20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:41:53.452256 24117 catalog_manager.cc:1259] Loaded cluster ID: 05c30684c8194129805b9367f5b8852e
I20260810 23:41:53.452374 24117 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: loading cluster ID for follower catalog manager: success
I20260810 23:41:53.455143 24117 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: acquiring CA information for follower catalog manager: success
I20260810 23:41:53.457161 24117 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf52c4e1d48748338c12a318d4bd11b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:41:53.559222 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 23654
I20260810 23:41:53.560714 23760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:53.672065 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 23654
I20260810 23:41:53.700520 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 23779
I20260810 23:41:53.701727 23892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:53.811301 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 23779
W20260810 23:41:53.834558 23954 connection.cc:535] server connection from 127.0.0.1:50268 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:41:53.834853 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 23906
I20260810 23:41:53.836345 24035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:53.952396 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 23906
2026-08-10T23:41:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4841 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:41:54 dist-test-slave-pww0 krb5kdc[24196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:41:54 dist-test-slave-pww0 krb5kdc[24196](info): set up 2 sockets
Aug 10 23:41:54 dist-test-slave-pww0 krb5kdc[24196](info): commencing operation
krb5kdc: starting...
W20260810 23:41:55.260649 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:41:55 dist-test-slave-pww0 krb5kdc[24196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:41:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:41:55.563388 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:34101
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:34101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:55.910326 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:55.910672 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:55.910806 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:55.918292 24217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:41:55.918447 24217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:55.918525 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:55.918599 24217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:41:55.918658 24217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:41:55.929028 24217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:34101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:34101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:55.930462 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:55.931833 24217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:55.944347 24224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:55.944306 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:55.944267 24230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:55.945668 24217 server_base.cc:1034] running on GCE node
I20260810 23:41:55.946563 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:55.948258 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:55.949514 24217 hybrid_clock.cc:648] HybridClock initialized: now 1786405315949456 us; error 51 us; skew 500 ppm
I20260810 23:41:55.949922 24217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:55 dist-test-slave-pww0 krb5kdc[24196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405315, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:55.953270 24217 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:41:55.955035 24217 webserver.cc:460] Webserver started at http://127.30.33.62:46257/ using document root <none> and password file <none>
I20260810 23:41:55.956007 24217 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:55.956178 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:55.956526 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:41:55.959864 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "df9a5a5c68914e17b81b19cac1bc161b"
format_stamp: "Formatted at 2026-08-10 23:41:55 on dist-test-slave-pww0"
server_key: "a5df0835790c7e1d148e06b426837bef"
server_key_iv: "267a001a156c43080f24f0b5bab411e6"
server_key_version: "encryptionkey@0"
I20260810 23:41:55.960692 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "df9a5a5c68914e17b81b19cac1bc161b"
format_stamp: "Formatted at 2026-08-10 23:41:55 on dist-test-slave-pww0"
server_key: "a5df0835790c7e1d148e06b426837bef"
server_key_iv: "267a001a156c43080f24f0b5bab411e6"
server_key_version: "encryptionkey@0"
I20260810 23:41:55.966514 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260810 23:41:55.970498 24246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:55.972090 24217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:41:55.972289 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "df9a5a5c68914e17b81b19cac1bc161b"
format_stamp: "Formatted at 2026-08-10 23:41:55 on dist-test-slave-pww0"
server_key: "a5df0835790c7e1d148e06b426837bef"
server_key_iv: "267a001a156c43080f24f0b5bab411e6"
server_key_version: "encryptionkey@0"
I20260810 23:41:55.972506 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:55.989130 24217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:55.994132 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:55.994429 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:56.013315 24217 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:34101
I20260810 23:41:56.013350 24345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:34101 every 8 connection(s)
I20260810 23:41:56.014787 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:41:56.018944 24346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:56.023298 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 24217
I20260810 23:41:56.023590 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:41:56.024430 30852 external_mini_cluster.cc:1428] Setting key 8ff5221f532654373ea42c9e0ca951c5
I20260810 23:41:56.031414 24346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b: Bootstrap starting.
I20260810 23:41:56.035349 24346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:56.036618 24346 log.cc:824] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:56.040365 24346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b: No bootstrap required, opened a new log
Aug 10 23:41:56 dist-test-slave-pww0 krb5kdc[24196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:56.048457 24346 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9a5a5c68914e17b81b19cac1bc161b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34101 } }
I20260810 23:41:56.048910 24346 raft_consensus.cc:374] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:56.049033 24346 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9a5a5c68914e17b81b19cac1bc161b, State: Initialized, Role: FOLLOWER
I20260810 23:41:56.049626 24346 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9a5a5c68914e17b81b19cac1bc161b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34101 } }
I20260810 23:41:56.049830 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:56.026462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.30.33.62:34101)
0810 23:41:56.027296 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:56.027309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:56.027354 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:41:56.029764 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:56.029792 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:56.030206 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:56.030593 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:56.032552 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:56.033961 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:56.035911 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:56.036649 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:56.036836 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:56.041845 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:56.041855 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:56.041876 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:56.041919 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:56.045215 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:56.046004 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:56.046012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:56.046023 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:56.046219 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:56.046635 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:56.046639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:56.046644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:56.047274 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:56.047605 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:56.048114 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 23:41:56.048559 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":383,"threads_started":1}
I20260810 23:41:56.049889 24346 raft_consensus.cc:388] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:41:56.050032 24346 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:41:56.050240 24346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:56.053236 24346 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9a5a5c68914e17b81b19cac1bc161b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34101 } }
I20260810 23:41:56.053808 24346 leader_election.cc:302] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df9a5a5c68914e17b81b19cac1bc161b; no voters:
I20260810 23:41:56.054028 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:41:56.054481 24346 leader_election.cc:288] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:41:56.054590 24355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:41:56.056808 24355 raft_consensus.cc:686] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [term 1 LEADER]: Becoming Leader. State: Replica: df9a5a5c68914e17b81b19cac1bc161b, State: Running, Role: LEADER
I20260810 23:41:56.057024 24346 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:41:56.057490 24355 consensus_queue.cc:237] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9a5a5c68914e17b81b19cac1bc161b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34101 } }
I20260810 23:41:56.060765 24356 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df9a5a5c68914e17b81b19cac1bc161b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9a5a5c68914e17b81b19cac1bc161b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34101 } } }
I20260810 23:41:56.060797 24361 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [sys.catalog]: SysCatalogTable state changed. Reason: New leader df9a5a5c68914e17b81b19cac1bc161b. Latest consensus state: current_term: 1 leader_uuid: "df9a5a5c68914e17b81b19cac1bc161b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9a5a5c68914e17b81b19cac1bc161b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 34101 } } }
I20260810 23:41:56.061226 24361 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:56.061232 24356 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:56.062999 24374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:41:56.068373 24374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:41:56.078665 24374 catalog_manager.cc:1347] Generated new cluster ID: 3fa946be9ab84c50b4551859d6c95169
I20260810 23:41:56.078800 24374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:41:56.096511 24374 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:41:56.098061 24374 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:41:56.110966 24374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df9a5a5c68914e17b81b19cac1bc161b: Generated new TSK 0
I20260810 23:41:56.111941 24374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:41:56.167094 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 24217
I20260810 23:41:56.168545 24341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:41:56.268728 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 24217
2026-08-10T23:41:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2311 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:41:56 dist-test-slave-pww0 krb5kdc[24400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:41:56 dist-test-slave-pww0 krb5kdc[24400](info): set up 2 sockets
Aug 10 23:41:56 dist-test-slave-pww0 krb5kdc[24400](info): commencing operation
krb5kdc: starting...
W20260810 23:41:57.572784 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:41:57 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:41:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:41:57.880071 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41349
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:58.230429 24430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:58.230754 24430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:58.230876 24430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:58.238312 24430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:41:58.238452 24430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:58.238510 24430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:58.238632 24430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:41:58.238686 24430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:41:58.249007 24430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:58.250489 24430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:58.251915 24430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:58.264441 24438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:58.264473 24440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:58.264505 24443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:58.265542 24430 server_base.cc:1034] running on GCE node
I20260810 23:41:58.266456 24430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:58.268126 24430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:58.269371 24430 hybrid_clock.cc:648] HybridClock initialized: now 1786405318269316 us; error 45 us; skew 500 ppm
I20260810 23:41:58.269773 24430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:58 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405318, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:58.273066 24430 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:41:58.274746 24430 webserver.cc:460] Webserver started at http://127.30.33.62:38839/ using document root <none> and password file <none>
I20260810 23:41:58.275445 24430 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:58.275630 24430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:58.275920 24430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:41:58.279035 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "ee93b0ea15964746b0cc39308d6893ac"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "4ef776f85893f212487248342f9ae21f"
server_key_iv: "60c2b311a78d8c3c2d898d531b91cbb3"
server_key_version: "encryptionkey@0"
I20260810 23:41:58.279870 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "ee93b0ea15964746b0cc39308d6893ac"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "4ef776f85893f212487248342f9ae21f"
server_key_iv: "60c2b311a78d8c3c2d898d531b91cbb3"
server_key_version: "encryptionkey@0"
I20260810 23:41:58.285398 24430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:41:58.289383 24457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:58.290938 24430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:41:58.291126 24430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "ee93b0ea15964746b0cc39308d6893ac"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "4ef776f85893f212487248342f9ae21f"
server_key_iv: "60c2b311a78d8c3c2d898d531b91cbb3"
server_key_version: "encryptionkey@0"
I20260810 23:41:58.291334 24430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:58.322480 24430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:58.327198 24430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:58.327515 24430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:58.346895 24430 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:41349
I20260810 23:41:58.346948 24545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:41349 every 8 connection(s)
I20260810 23:41:58.348421 24430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:41:58.351703 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 24430
I20260810 23:41:58.351954 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:41:58.352804 30852 external_mini_cluster.cc:1428] Setting key 64dd5cd272b9d8386258621e05b0c835
I20260810 23:41:58.353118 24546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:41:58.368652 24546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Bootstrap starting.
Aug 10 23:41:58 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:58.372318 24546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Neither blocks nor log segments found. Creating new log.
I20260810 23:41:58.373533 24546 log.cc:824] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Log is configured to *not* fsync() on all Append() calls
I20260810 23:41:58.376757 24546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: No bootstrap required, opened a new log
I20260810 23:41:58.378408 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:58.355142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.30.33.62:41349)
0810 23:41:58.356044 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:58.356056 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:58.356103 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:41:58.358077 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:58.358102 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:58.358567 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:58.358993 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:58.361010 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:58.362390 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:58.364355 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:58.365157 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:58.365323 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:58.370488 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:58.370498 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:58.370515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:58.370569 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:58.373881 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:58.374732 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:58.374740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:58.374748 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:58.374944 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:58.375372 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:58.375376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:58.375378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:58.376020 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:58.376384 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:58.376856 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:41:58.377342 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":404,"threads_started":1}
I20260810 23:41:58.385006 24546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:41:58.385437 24546 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:41:58.385540 24546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee93b0ea15964746b0cc39308d6893ac, State: Initialized, Role: FOLLOWER
I20260810 23:41:58.386204 24546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:41:58.386502 24546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:41:58.386664 24546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:41:58.386865 24546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:41:58.389302 24546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:41:58.389801 24546 leader_election.cc:302] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee93b0ea15964746b0cc39308d6893ac; no voters:
I20260810 23:41:58.390394 24546 leader_election.cc:288] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:41:58.390514 24551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:41:58.391165 24551 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 LEADER]: Becoming Leader. State: Replica: ee93b0ea15964746b0cc39308d6893ac, State: Running, Role: LEADER
I20260810 23:41:58.391749 24551 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:41:58.392839 24546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:41:58.394711 24555 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee93b0ea15964746b0cc39308d6893ac. Latest consensus state: current_term: 1 leader_uuid: "ee93b0ea15964746b0cc39308d6893ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } } }
I20260810 23:41:58.394683 24553 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee93b0ea15964746b0cc39308d6893ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } } }
I20260810 23:41:58.395277 24555 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:58.395288 24553 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:41:58.397866 24563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:41:58.403810 24563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:41:58.412982 24563 catalog_manager.cc:1347] Generated new cluster ID: 9af2e1d7d5094432a86aa32a38dc991e
I20260810 23:41:58.413105 24563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:41:58.420948 24563 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:41:58.422366 24563 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:41:58.429836 24563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Generated new TSK 0
I20260810 23:41:58.430744 24563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:41:58.503039 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41349
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:58.906880 24597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:58.907212 24597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:58.907339 24597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:58.915165 24597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:58.915304 24597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:58.915488 24597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:41:58.925848 24597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:58.927275 24597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:58.928746 24597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:58.942466 24604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:58.942502 24605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:58.942534 24610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:58.943528 24597 server_base.cc:1034] running on GCE node
I20260810 23:41:58.944203 24597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:58.945245 24597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:58.946485 24597 hybrid_clock.cc:648] HybridClock initialized: now 1786405318946418 us; error 53 us; skew 500 ppm
I20260810 23:41:58.946846 24597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:58 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405318, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:58.950371 24597 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:41:58.952134 24597 webserver.cc:460] Webserver started at http://127.30.33.1:33391/ using document root <none> and password file <none>
I20260810 23:41:58.952909 24597 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:58.953029 24597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:58.953419 24597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:41:58.956960 24597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "89d70d0a2af04d79b0d1b7f9d943f318"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "565b9f636ffb1ee75c1746a294c33f9f"
server_key_iv: "7ae9aea8b6b3355fa50afaf8142a86b0"
server_key_version: "encryptionkey@0"
I20260810 23:41:58.957823 24597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "89d70d0a2af04d79b0d1b7f9d943f318"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "565b9f636ffb1ee75c1746a294c33f9f"
server_key_iv: "7ae9aea8b6b3355fa50afaf8142a86b0"
server_key_version: "encryptionkey@0"
I20260810 23:41:58.964056 24597 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 23:41:58.968037 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:58.969671 24597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:41:58.969870 24597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "89d70d0a2af04d79b0d1b7f9d943f318"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "565b9f636ffb1ee75c1746a294c33f9f"
server_key_iv: "7ae9aea8b6b3355fa50afaf8142a86b0"
server_key_version: "encryptionkey@0"
I20260810 23:41:58.970095 24597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:59.001809 24597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:59.007162 24597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:59.007547 24597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:59.008702 24597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:59.010434 24597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:59.010542 24597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:59.010679 24597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:59.010736 24597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:59.044945 24597 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:42071
I20260810 23:41:59.044993 24835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:42071 every 8 connection(s)
I20260810 23:41:59.046339 24597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:41:59.051352 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 24597
I20260810 23:41:59.051612 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:41:59.052315 30852 external_mini_cluster.cc:1428] Setting key 7c71b54945d134cd763d6c88bee915b5
Aug 10 23:41:59 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405318, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:59.069720 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:59.049091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:49015 (local address 127.30.33.62:41349)
0810 23:41:59.049613 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:59.049617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:59.050497 (+ 880us) server_negotiation.cc:407] Connection header received
0810 23:41:59.052625 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:59.052631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:59.052742 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:59.052846 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:59.054834 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.055634 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.057305 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.057725 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.057848 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:59.063331 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:59.063337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:59.063340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:59.063384 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:59.066066 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:59.067039 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:59.067044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:59.067046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:59.067210 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:59.067775 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:59.067779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:59.067781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:59.068080 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:59.068239 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:59.069063 (+ 824us) server_negotiation.cc:299] Negotiation successful
0810 23:41:59.069339 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":267,"threads_started":1}
I20260810 23:41:59.070823 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:59.049500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:49015)
0810 23:41:59.050304 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:59.050355 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:59.052284 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:59.053147 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:59.053157 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:59.053920 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:59.054640 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:59.054661 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.055816 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.055820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:59.057138 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:59.057153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.057985 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.057993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:59.058249 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:59.059049 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:59.059077 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:59.062988 (+ 3911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:59.066312 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:59.066322 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:59.066339 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:59.066882 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:59.067349 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:59.067354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:59.067356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:59.067650 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:59.068236 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:59.068245 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:59.068650 (+ 405us) client_negotiation.cc:769] Sending connection context
0810 23:41:59.068914 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 23:41:59.069539 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":321,"threads_started":1}
I20260810 23:41:59.072791 24836 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41349
I20260810 23:41:59.073182 24836 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:59.074256 24836 heartbeater.cc:511] Master 127.30.33.62:41349 requested a full tablet report, sending...
I20260810 23:41:59.077193 24487 ts_manager.cc:194] Registered new tserver with Master: 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071)
I20260810 23:41:59.079222 24487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:49015
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:41:59.123224 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41349
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:41:59.501937 24853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:41:59.502239 24853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:41:59.502367 24853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:41:59.509939 24853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:41:59.510063 24853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:41:59.510218 24853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:41:59.520344 24853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:41:59.521724 24853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:41:59.523077 24853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:41:59.537581 24863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:59.537581 24866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:41:59.538151 24864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:41:59.539376 24853 server_base.cc:1034] running on GCE node
I20260810 23:41:59.540148 24853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:41:59.541234 24853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:41:59.542474 24853 hybrid_clock.cc:648] HybridClock initialized: now 1786405319542412 us; error 51 us; skew 500 ppm
I20260810 23:41:59.542845 24853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:41:59 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405319, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:41:59.546424 24853 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:41:59.548159 24853 webserver.cc:460] Webserver started at http://127.30.33.2:35229/ using document root <none> and password file <none>
I20260810 23:41:59.548921 24853 fs_manager.cc:359] Metadata directory not provided
I20260810 23:41:59.549046 24853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:41:59.549394 24853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:41:59.552729 24853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "1c5b9ebf2e384974b8ba060d72a49696"
format_stamp: "Formatted at 2026-08-10 23:41:59 on dist-test-slave-pww0"
server_key: "c719951ad3a0d446e3271588e5f1d296"
server_key_iv: "d3ccc637868e3a7eb544498e71483770"
server_key_version: "encryptionkey@0"
I20260810 23:41:59.553634 24853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "1c5b9ebf2e384974b8ba060d72a49696"
format_stamp: "Formatted at 2026-08-10 23:41:59 on dist-test-slave-pww0"
server_key: "c719951ad3a0d446e3271588e5f1d296"
server_key_iv: "d3ccc637868e3a7eb544498e71483770"
server_key_version: "encryptionkey@0"
I20260810 23:41:59.559630 24853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:41:59.563807 24882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:59.565517 24853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:41:59.565733 24853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "1c5b9ebf2e384974b8ba060d72a49696"
format_stamp: "Formatted at 2026-08-10 23:41:59 on dist-test-slave-pww0"
server_key: "c719951ad3a0d446e3271588e5f1d296"
server_key_iv: "d3ccc637868e3a7eb544498e71483770"
server_key_version: "encryptionkey@0"
I20260810 23:41:59.565953 24853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:41:59.587396 24853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:41:59.591856 24853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:41:59.592152 24853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:41:59.593269 24853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:41:59.594877 24853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:41:59.594955 24853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:59.595081 24853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:41:59.595131 24853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:41:59.627103 24853 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:42531
I20260810 23:41:59.627148 25086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:42531 every 8 connection(s)
I20260810 23:41:59.628489 24853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:41:59.631861 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 24853
I20260810 23:41:59.632068 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:41:59.632589 30852 external_mini_cluster.cc:1428] Setting key ed33bf30f98afe6cc90d3fa2cfdbf8bc
Aug 10 23:41:59 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405319, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:41:59.651952 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:59.631513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:38145 (local address 127.30.33.62:41349)
0810 23:41:59.632039 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 23:41:59.632046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:41:59.633231 (+ 1185us) server_negotiation.cc:407] Connection header received
0810 23:41:59.635656 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:41:59.635663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:41:59.635761 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:41:59.635870 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:41:59.637581 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.638558 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.640544 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.641141 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.641284 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:59.645345 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:41:59.645353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:41:59.645357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:41:59.645403 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:41:59.648225 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:59.649214 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:59.649219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:59.649222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:59.649360 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:41:59.649864 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:41:59.649866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:41:59.649868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:41:59.650151 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:41:59.650282 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:41:59.651320 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0810 23:41:59.651604 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
I20260810 23:41:59.653338 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:41:59.632086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:38145)
0810 23:41:59.633024 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0810 23:41:59.633081 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:41:59.635311 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:41:59.636151 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:41:59.636161 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:41:59.636748 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:41:59.637386 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:59.637407 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.638751 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.638758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:59.640365 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:41:59.640383 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:41:59.641407 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:41:59.641413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:41:59.641696 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:41:59.642447 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:41:59.642478 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:41:59.645035 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:41:59.648404 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:59.648416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:59.648436 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:59.649090 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:59.649468 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:41:59.649473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:41:59.649476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:41:59.649761 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:41:59.650290 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:41:59.650298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:41:59.650810 (+ 512us) client_negotiation.cc:769] Sending connection context
0810 23:41:59.651196 (+ 386us) client_negotiation.cc:240] Negotiation successful
0810 23:41:59.651842 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":316,"threads_started":1}
I20260810 23:41:59.654990 25087 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41349
I20260810 23:41:59.655364 25087 heartbeater.cc:464] Registering TS with master...
I20260810 23:41:59.656433 25087 heartbeater.cc:511] Master 127.30.33.62:41349 requested a full tablet report, sending...
I20260810 23:41:59.658165 24487 ts_manager.cc:194] Registered new tserver with Master: 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531)
I20260810 23:41:59.659020 24487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:38145
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:41:59.702483 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41349
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:00.056730 25097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:00.057077 25097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:00.057204 25097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:00.065258 25097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:00.065409 25097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:00.065567 25097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:42:00.076107 25097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:00.077565 25097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:00.079015 25097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:42:00.083359 24836 heartbeater.cc:503] Master 127.30.33.62:41349 was elected leader, sending a full tablet report...
W20260810 23:42:00.093456 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:00.093386 25106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:00.093402 25109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:00.095057 25097 server_base.cc:1034] running on GCE node
I20260810 23:42:00.095922 25097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:00.097046 25097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:00.098284 25097 hybrid_clock.cc:648] HybridClock initialized: now 1786405320098229 us; error 53 us; skew 500 ppm
I20260810 23:42:00.098650 25097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:00 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405320, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:00.102000 25097 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:42:00.103821 25097 webserver.cc:460] Webserver started at http://127.30.33.3:42975/ using document root <none> and password file <none>
I20260810 23:42:00.104568 25097 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:00.104696 25097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:00.105002 25097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:42:00.108172 25097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "497cfb3944fc4afab27988eabe2836bc"
format_stamp: "Formatted at 2026-08-10 23:42:00 on dist-test-slave-pww0"
server_key: "d76260f5de69dfc08ba98a306d3242a1"
server_key_iv: "47a542198c767a37a75c2dbc64921763"
server_key_version: "encryptionkey@0"
I20260810 23:42:00.109047 25097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "497cfb3944fc4afab27988eabe2836bc"
format_stamp: "Formatted at 2026-08-10 23:42:00 on dist-test-slave-pww0"
server_key: "d76260f5de69dfc08ba98a306d3242a1"
server_key_iv: "47a542198c767a37a75c2dbc64921763"
server_key_version: "encryptionkey@0"
I20260810 23:42:00.114725 25097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:42:00.118682 25125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:00.120244 25097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:42:00.120431 25097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "497cfb3944fc4afab27988eabe2836bc"
format_stamp: "Formatted at 2026-08-10 23:42:00 on dist-test-slave-pww0"
server_key: "d76260f5de69dfc08ba98a306d3242a1"
server_key_iv: "47a542198c767a37a75c2dbc64921763"
server_key_version: "encryptionkey@0"
I20260810 23:42:00.120642 25097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:00.144337 25097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:00.148733 25097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:00.149035 25097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:00.150266 25097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:00.151875 25097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:00.151970 25097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:00.152100 25097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:00.152158 25097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:00.184443 25097 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:41719
I20260810 23:42:00.184496 25328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:41719 every 8 connection(s)
I20260810 23:42:00.185864 25097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:42:00.194617 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 25097
I20260810 23:42:00.194839 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:42:00.195313 30852 external_mini_cluster.cc:1428] Setting key fd484adff443f5eaa183a01a4718688b
Aug 10 23:42:00 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405320, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:00.207355 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.188669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:49309 (local address 127.30.33.62:41349)
0810 23:42:00.189129 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.189133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.190097 (+ 964us) server_negotiation.cc:407] Connection header received
0810 23:42:00.192061 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.192067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.192172 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.192275 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:00.194047 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.194976 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.196956 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.197513 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.197631 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.201356 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:00.201361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:00.201365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:00.201406 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:00.204012 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.204831 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.204836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.204839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.204980 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.205451 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.205455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.205457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.205763 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:00.205922 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.206692 (+ 770us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.206946 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":221,"threads_started":1}
I20260810 23:42:00.208397 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.189176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:49309)
0810 23:42:00.189925 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:00.189969 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:00.191754 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:00.192611 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:00.192620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:00.193172 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:00.193853 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.193878 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.195176 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.195180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.196813 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.196827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.197766 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.197775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.198057 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.198739 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:00.198763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:00.201057 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:00.204184 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:00.204194 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:00.204211 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:00.204686 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:00.205097 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:00.205101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:00.205105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:00.205326 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:00.205903 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:00.205912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:00.206303 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 23:42:00.206556 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 23:42:00.207142 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":277,"threads_started":1}
I20260810 23:42:00.209992 25335 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41349
I20260810 23:42:00.210340 25335 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:00.211112 25335 heartbeater.cc:511] Master 127.30.33.62:41349 requested a full tablet report, sending...
I20260810 23:42:00.212615 24487 ts_manager.cc:194] Registered new tserver with Master: 497cfb3944fc4afab27988eabe2836bc (127.30.33.3:41719)
I20260810 23:42:00.213490 24487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:49309
I20260810 23:42:00.223954 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:42:00.241489 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.229072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.30.33.62:41349)
0810 23:42:00.229299 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.229304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.229676 (+ 372us) server_negotiation.cc:407] Connection header received
0810 23:42:00.229839 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.229843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.229914 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.230003 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:00.231501 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.232259 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.233646 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.234051 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.234146 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.236618 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:00.236624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:00.236627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:00.236664 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:00.239085 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.239808 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.239812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.239814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.239947 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.240257 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.240260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.240263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.240602 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:00.240803 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.241048 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.241249 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:42:00.245857 24487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:42:00.253027 24487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:42:00.278771 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.266869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.30.33.2:42531)
0810 23:42:00.268438 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.268449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.269125 (+ 676us) server_negotiation.cc:407] Connection header received
0810 23:42:00.269265 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.269273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.269636 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.269847 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:00.270897 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.272669 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.275830 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.277331 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.277474 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.277689 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.278251 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.278436 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1387,"thread_start_us":1271,"threads_started":1}
I20260810 23:42:00.279285 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.269620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.30.33.3:41719)
0810 23:42:00.270636 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.270644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.270679 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:42:00.270786 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.270793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.271019 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.271901 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:00.272126 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.273436 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.276148 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.277170 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.277369 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.277672 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.278725 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.278937 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":566,"threads_started":1}
I20260810 23:42:00.279285 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.267299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.30.33.1:42071)
0810 23:42:00.269211 (+ 1912us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.269220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.269243 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:42:00.270437 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.270459 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.270767 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.271596 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:00.271854 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.273180 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.276112 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.277595 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.277725 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.277951 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.278750 (+ 799us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.278937 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":1547,"threads_started":1}
I20260810 23:42:00.279582 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.269262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:41719 (local address 127.0.0.1:48546)
0810 23:42:00.269847 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:00.269865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:00.269970 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:00.271241 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:00.271246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:00.271271 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:00.271758 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.271770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.273594 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.273599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.275950 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.275962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.277398 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.277406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.277616 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.277684 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 23:42:00.277845 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:42:00.278083 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":50,"thread_start_us":313,"threads_started":1}
I20260810 23:42:00.279996 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.266817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42531 (local address 127.0.0.1:40944)
0810 23:42:00.268826 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:00.268847 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:00.268984 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:00.269962 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:00.269969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:00.269994 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:00.270470 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.270490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.272823 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.272829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.275634 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.275651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.277850 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.277857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.278017 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.278058 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:42:00.278153 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:42:00.278294 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"thread_start_us":1717,"threads_started":1}
I20260810 23:42:00.280048 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.266805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:59596)
0810 23:42:00.267647 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:00.267693 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:00.267864 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:00.271000 (+ 3136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:00.271005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:00.271031 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:00.271565 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.271582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.273317 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.273321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.275886 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.275906 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.277902 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.277910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.278077 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.278103 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:42:00.278191 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:42:00.278337 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":93,"thread_start_us":747,"threads_started":1}
I20260810 23:42:00.283818 24716 tablet_service.cc:1467] Processing CreateTablet for tablet c505b90f7c6247e480878c9dcdf1f2dc (DEFAULT_TABLE table=test-table [id=ec15e07fad8c4dc38f4aa15921167b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:00.284085 24967 tablet_service.cc:1467] Processing CreateTablet for tablet c505b90f7c6247e480878c9dcdf1f2dc (DEFAULT_TABLE table=test-table [id=ec15e07fad8c4dc38f4aa15921167b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:00.284086 25204 tablet_service.cc:1467] Processing CreateTablet for tablet c505b90f7c6247e480878c9dcdf1f2dc (DEFAULT_TABLE table=test-table [id=ec15e07fad8c4dc38f4aa15921167b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:00.285490 24716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c505b90f7c6247e480878c9dcdf1f2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:00.285495 25204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c505b90f7c6247e480878c9dcdf1f2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:00.285501 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c505b90f7c6247e480878c9dcdf1f2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:00.297943 25373 tablet_bootstrap.cc:492] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Bootstrap starting.
I20260810 23:42:00.300956 25374 tablet_bootstrap.cc:492] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Bootstrap starting.
I20260810 23:42:00.303884 25373 tablet_bootstrap.cc:654] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:00.304719 25374 tablet_bootstrap.cc:654] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:00.305397 25373 log.cc:824] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:00.305969 25374 log.cc:824] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:00.306833 25375 tablet_bootstrap.cc:492] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Bootstrap starting.
I20260810 23:42:00.309717 25373 tablet_bootstrap.cc:492] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: No bootstrap required, opened a new log
I20260810 23:42:00.310034 25373 ts_tablet_manager.cc:1397] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260810 23:42:00.311355 25375 tablet_bootstrap.cc:654] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:00.312762 25375 log.cc:824] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:00.315752 25375 tablet_bootstrap.cc:492] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: No bootstrap required, opened a new log
I20260810 23:42:00.315984 25374 tablet_bootstrap.cc:492] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: No bootstrap required, opened a new log
I20260810 23:42:00.316151 25375 ts_tablet_manager.cc:1397] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 23:42:00.316279 25374 ts_tablet_manager.cc:1397] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260810 23:42:00.320726 25373 raft_consensus.cc:348] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.321244 25373 raft_consensus.cc:374] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:00.321398 25373 raft_consensus.cc:729] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d70d0a2af04d79b0d1b7f9d943f318, State: Initialized, Role: FOLLOWER
I20260810 23:42:00.322121 25373 consensus_queue.cc:260] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.324236 25373 ts_tablet_manager.cc:1428] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260810 23:42:00.325119 25375 raft_consensus.cc:348] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.325611 25375 raft_consensus.cc:374] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:00.325737 25375 raft_consensus.cc:729] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5b9ebf2e384974b8ba060d72a49696, State: Initialized, Role: FOLLOWER
I20260810 23:42:00.326364 25375 consensus_queue.cc:260] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.326467 25374 raft_consensus.cc:348] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.327075 25374 raft_consensus.cc:374] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:00.327234 25374 raft_consensus.cc:729] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497cfb3944fc4afab27988eabe2836bc, State: Initialized, Role: FOLLOWER
I20260810 23:42:00.327590 25087 heartbeater.cc:503] Master 127.30.33.62:41349 was elected leader, sending a full tablet report...
I20260810 23:42:00.328055 25374 consensus_queue.cc:260] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.328253 25375 ts_tablet_manager.cc:1428] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260810 23:42:00.329216 25335 heartbeater.cc:503] Master 127.30.33.62:41349 was elected leader, sending a full tablet report...
I20260810 23:42:00.331383 25374 ts_tablet_manager.cc:1428] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260810 23:42:00.381011 25088 tablet.cc:2369] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:42:00.437529 25336 tablet.cc:2369] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:42:00.549120 24840 tablet.cc:2369] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:42:00.622931 25388 raft_consensus.cc:482] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:42:00.623293 25388 raft_consensus.cc:504] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.625200 25388 leader_election.cc:288] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531), 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071)
I20260810 23:42:00.631891 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.625447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42531 (local address 127.30.33.3:57643)
0810 23:42:00.625768 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:00.625789 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:00.625937 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:00.626284 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:00.626288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:00.626316 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:00.626502 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.626511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.627793 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.627798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.629303 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.629317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.630945 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.630952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.631253 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.631293 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:42:00.631410 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:42:00.631594 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":24}
I20260810 23:42:00.631888 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.625575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:57643 (local address 127.30.33.2:42531)
0810 23:42:00.625803 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.625809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.625905 (+ 96us) server_negotiation.cc:407] Connection header received
0810 23:42:00.626071 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.626075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.626137 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.626213 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:00.626650 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.627635 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.629460 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.630743 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.630886 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.630986 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.631447 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.631612 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:42:00.633302 25005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c505b90f7c6247e480878c9dcdf1f2dc" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5b9ebf2e384974b8ba060d72a49696" is_pre_election: true
I20260810 23:42:00.633756 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.625742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:60493 (local address 127.30.33.1:42071)
0810 23:42:00.625973 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.625979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.627327 (+ 1348us) server_negotiation.cc:407] Connection header received
0810 23:42:00.627481 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.627487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.627589 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.627687 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:00.628419 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.629490 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.631675 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.632844 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.632970 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.633074 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.633367 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.633498 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:42:00.633817 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.625735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.30.33.3:60493)
0810 23:42:00.627191 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:00.627219 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:00.627340 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:00.627729 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:00.627733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:00.627759 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:00.628222 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.628234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.629651 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.629657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.631491 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:00.631507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.633043 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.633050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:00.633204 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.633238 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:42:00.633317 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:42:00.633440 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":28,"thread_start_us":425,"threads_started":1}
I20260810 23:42:00.633903 25005 raft_consensus.cc:2454] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 0.
I20260810 23:42:00.634667 25126 leader_election.cc:302] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5b9ebf2e384974b8ba060d72a49696, 497cfb3944fc4afab27988eabe2836bc; no voters:
I20260810 23:42:00.634943 24756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c505b90f7c6247e480878c9dcdf1f2dc" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" is_pre_election: true
I20260810 23:42:00.635144 25388 raft_consensus.cc:2781] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:42:00.635290 25388 raft_consensus.cc:482] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:42:00.635381 25388 raft_consensus.cc:3037] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:00.635504 24756 raft_consensus.cc:2454] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 0.
I20260810 23:42:00.637951 25388 raft_consensus.cc:504] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.638651 25388 leader_election.cc:288] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 election: Requested vote from peers 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531), 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071)
I20260810 23:42:00.639211 25005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c505b90f7c6247e480878c9dcdf1f2dc" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5b9ebf2e384974b8ba060d72a49696"
I20260810 23:42:00.639283 24756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c505b90f7c6247e480878c9dcdf1f2dc" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d70d0a2af04d79b0d1b7f9d943f318"
I20260810 23:42:00.639501 25005 raft_consensus.cc:3037] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:00.639518 24756 raft_consensus.cc:3037] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:00.643009 24756 raft_consensus.cc:2454] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 1.
I20260810 23:42:00.643000 25005 raft_consensus.cc:2454] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 1.
I20260810 23:42:00.643607 25126 leader_election.cc:302] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5b9ebf2e384974b8ba060d72a49696, 497cfb3944fc4afab27988eabe2836bc; no voters:
I20260810 23:42:00.644001 25388 raft_consensus.cc:2781] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:42:00.644497 25388 raft_consensus.cc:686] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 1 LEADER]: Becoming Leader. State: Replica: 497cfb3944fc4afab27988eabe2836bc, State: Running, Role: LEADER
I20260810 23:42:00.645017 25388 consensus_queue.cc:237] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } }
I20260810 23:42:00.650357 24485 catalog_manager.cc:5495] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 497cfb3944fc4afab27988eabe2836bc (127.30.33.3). New cstate: current_term: 1 leader_uuid: "497cfb3944fc4afab27988eabe2836bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } health_report { overall_health: HEALTHY } } }
I20260810 23:42:00.738152 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.732523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.30.33.3:41719)
0810 23:42:00.732771 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.732777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.732799 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:42:00.732955 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.732959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.733073 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.733216 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:42:00.733596 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.734351 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.735892 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.736295 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.736427 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.736989 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:42:00.737539 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:42:00.737681 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.737789 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.737879 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:42:00.746908 25005 raft_consensus.cc:1261] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Refusing update from remote peer 497cfb3944fc4afab27988eabe2836bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:00.746906 24756 raft_consensus.cc:1261] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Refusing update from remote peer 497cfb3944fc4afab27988eabe2836bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:00.747915 25392 consensus_queue.cc:1035] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:00.748288 25388 consensus_queue.cc:1035] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:00.759588 25395 mvcc.cc:205] Tried to move back new op lower bound from 7317116193768812544 to 7317116193365966848. Current Snapshot: MvccSnapshot[applied={T|T < 7317116193768812544}]
I20260810 23:42:00.764326 25396 mvcc.cc:205] Tried to move back new op lower bound from 7317116193768812544 to 7317116193365966848. Current Snapshot: MvccSnapshot[applied={T|T < 7317116193768812544}]
I20260810 23:42:00.775445 24485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:00.775843 24485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:00.780148 24485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Sending DeleteTablet for 3 replicas of tablet c505b90f7c6247e480878c9dcdf1f2dc
I20260810 23:42:00.781627 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet c505b90f7c6247e480878c9dcdf1f2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:00 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:40944
I20260810 23:42:00.781627 24716 tablet_service.cc:1514] Processing DeleteTablet for tablet c505b90f7c6247e480878c9dcdf1f2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:00 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:59596
I20260810 23:42:00.781734 25204 tablet_service.cc:1514] Processing DeleteTablet for tablet c505b90f7c6247e480878c9dcdf1f2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:00 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:48546
I20260810 23:42:00.783330 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 24597
I20260810 23:42:00.794740 25419 tablet_replica.cc:330] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: stopping tablet replica
I20260810 23:42:00.795570 25419 raft_consensus.cc:2229] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:00.796453 25419 raft_consensus.cc:2258] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:00.797494 25414 tablet_replica.cc:330] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: stopping tablet replica
I20260810 23:42:00.798193 25414 raft_consensus.cc:2229] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:00.798751 25425 tablet_replica.cc:330] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: stopping tablet replica
I20260810 23:42:00.799275 25425 raft_consensus.cc:2229] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:42:00.800128 25419 ts_tablet_manager.cc:1905] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:00.800110 25425 raft_consensus.cc:2258] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:00.800881 25414 raft_consensus.cc:2258] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:00.802207 25425 ts_tablet_manager.cc:1905] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:00.804452 25414 ts_tablet_manager.cc:1905] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:00.809926 25425 ts_tablet_manager.cc:1918] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 10 23:42:00 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:42:00.810242 25425 log.cc:1196] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/c505b90f7c6247e480878c9dcdf1f2dc
I20260810 23:42:00.810930 25425 ts_tablet_manager.cc:1939] T c505b90f7c6247e480878c9dcdf1f2dc P 497cfb3944fc4afab27988eabe2836bc: Deleting consensus metadata
I20260810 23:42:00.813205 24461 catalog_manager.cc:4841] TS 497cfb3944fc4afab27988eabe2836bc (127.30.33.3:41719): tablet c505b90f7c6247e480878c9dcdf1f2dc (table test-table [id=ec15e07fad8c4dc38f4aa15921167b61]) successfully deleted
I20260810 23:42:00.814090 25419 ts_tablet_manager.cc:1918] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:00.814401 25419 log.cc:1196] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/c505b90f7c6247e480878c9dcdf1f2dc
I20260810 23:42:00.815053 25419 ts_tablet_manager.cc:1939] T c505b90f7c6247e480878c9dcdf1f2dc P 1c5b9ebf2e384974b8ba060d72a49696: Deleting consensus metadata
I20260810 23:42:00.817152 24458 catalog_manager.cc:4841] TS 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531): tablet c505b90f7c6247e480878c9dcdf1f2dc (table test-table [id=ec15e07fad8c4dc38f4aa15921167b61]) successfully deleted
I20260810 23:42:00.817726 25414 ts_tablet_manager.cc:1918] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:00.817890 25414 log.cc:1196] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/c505b90f7c6247e480878c9dcdf1f2dc
I20260810 23:42:00.818358 25414 ts_tablet_manager.cc:1939] T c505b90f7c6247e480878c9dcdf1f2dc P 89d70d0a2af04d79b0d1b7f9d943f318: Deleting consensus metadata
I20260810 23:42:00.820008 24459 catalog_manager.cc:4841] TS 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071): tablet c505b90f7c6247e480878c9dcdf1f2dc (table test-table [id=ec15e07fad8c4dc38f4aa15921167b61]) successfully deleted
I20260810 23:42:00.823025 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.784739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.30.33.1:42071)
0810 23:42:00.784931 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.784937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.797953 (+ 13016us) server_negotiation.cc:407] Connection header received
0810 23:42:00.798235 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.798240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.798374 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.798498 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:00.801245 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.802429 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.804639 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.805295 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.805424 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.811747 (+ 6323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:00.811770 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:00.811793 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:00.811846 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:00.817154 (+ 5308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.820606 (+ 3452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.820615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.820617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.820785 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.821320 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.821325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.821327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.821748 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:00.822000 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.822497 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.822716 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:42:00.823844 24825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:00.916914 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 24597
I20260810 23:42:00.954916 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 24853
Aug 10 23:42:00 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:42:00.974146 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:00.955826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.30.33.2:42531)
0810 23:42:00.956009 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:00.956014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:00.956052 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:42:00.956313 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:00.956319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:00.956393 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:00.956485 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:00.958221 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.959027 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.961119 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:00.961576 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:00.961698 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:00.967291 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:00.967311 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:00.967327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:00.967378 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:00.970697 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.971819 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.971829 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.971831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.971987 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:00.972414 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:00.972419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:00.972421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:00.972861 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:00.973161 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:00.973559 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 23:42:00.973809 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:42:00.974778 25079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:01.066148 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 24853
I20260810 23:42:01.097770 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 25097
Aug 10 23:42:01 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:42:01.116701 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:01.098742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.30.33.3:41719)
0810 23:42:01.098965 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:01.098971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:01.098996 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:42:01.099090 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:01.099095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:01.099195 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:01.099372 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:01.100880 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:01.102135 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:01.103662 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:01.104363 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:01.104518 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:01.109667 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:01.109686 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:01.109702 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:01.109750 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:01.113106 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:01.114192 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:01.114201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:01.114204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:01.114381 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:01.114887 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:01.114891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:01.114893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:01.115325 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:01.115656 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:01.116156 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 23:42:01.116348 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 23:42:01.117439 25319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:01.252297 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 25097
I20260810 23:42:01.285534 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 24430
I20260810 23:42:01.286978 24537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:01.404903 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 24430
I20260810 23:42:01.438187 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41349
--webserver_interface=127.30.33.62
--webserver_port=38839
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:01.795614 25437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:01.795986 25437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:01.796097 25437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:01.804087 25437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:42:01.804231 25437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:01.804278 25437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:01.804337 25437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:42:01.804381 25437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:42:01.814814 25437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:01.816196 25437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:01.817605 25437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:01.843694 25448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:01.843622 25452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:01.843647 25449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:01.844615 25437 server_base.cc:1034] running on GCE node
I20260810 23:42:01.845649 25437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:01.847429 25437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:01.848702 25437 hybrid_clock.cc:648] HybridClock initialized: now 1786405321848652 us; error 81 us; skew 500 ppm
I20260810 23:42:01.849144 25437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:01 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405321, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:01.859843 25437 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:42:01.862078 25437 webserver.cc:460] Webserver started at http://127.30.33.62:38839/ using document root <none> and password file <none>
I20260810 23:42:01.862977 25437 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:01.863155 25437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:01.870174 25437 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 23:42:01.873690 25470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:01.875277 25437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:42:01.875516 25437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "ee93b0ea15964746b0cc39308d6893ac"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "4ef776f85893f212487248342f9ae21f"
server_key_iv: "60c2b311a78d8c3c2d898d531b91cbb3"
server_key_version: "encryptionkey@0"
I20260810 23:42:01.876684 25437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:01.896063 25437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:01.901100 25437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:01.901419 25437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:01.921250 25437 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:41349
I20260810 23:42:01.921271 25569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:41349 every 8 connection(s)
I20260810 23:42:01.922861 25437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:42:01.930006 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 25437
I20260810 23:42:01.930955 25571 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:42:01.931064 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:42071
--local_ip_for_outbound_sockets=127.30.33.1
--tserver_master_addrs=127.30.33.62:41349
--webserver_port=33391
--webserver_interface=127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:42:01.935762 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Bootstrap starting.
I20260810 23:42:01.958154 25571 log.cc:824] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:01.971684 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:42:01.972255 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Bootstrap complete.
I20260810 23:42:01.981874 25571 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:01.982785 25571 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee93b0ea15964746b0cc39308d6893ac, State: Initialized, Role: FOLLOWER
I20260810 23:42:01.983564 25571 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:01.983847 25571 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:42:01.983978 25571 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:42:01.984155 25571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:42:01.988346 25571 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:01.988955 25571 leader_election.cc:302] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee93b0ea15964746b0cc39308d6893ac; no voters:
I20260810 23:42:01.989588 25571 leader_election.cc:288] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:42:01.989750 25577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:42:01.991089 25577 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 LEADER]: Becoming Leader. State: Replica: ee93b0ea15964746b0cc39308d6893ac, State: Running, Role: LEADER
I20260810 23:42:01.991825 25577 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:01.992190 25571 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:42:01.994428 25580 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee93b0ea15964746b0cc39308d6893ac. Latest consensus state: current_term: 2 leader_uuid: "ee93b0ea15964746b0cc39308d6893ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } } }
I20260810 23:42:01.994858 25580 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:01.995434 25579 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ee93b0ea15964746b0cc39308d6893ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } } }
I20260810 23:42:01.995780 25579 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:02.004707 25589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:42:02.015570 25589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:42:02.017539 25589 catalog_manager.cc:1259] Loaded cluster ID: 9af2e1d7d5094432a86aa32a38dc991e
I20260810 23:42:02.017776 25589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:42:02.021796 25589 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 23:42:02.041544 25589 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 23:42:02.045351 25615 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:42:02.056182 25589 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:42:02.058137 25589 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Loaded TSK: 0
I20260810 23:42:02.059991 25589 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:42:02.318259 25575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:02.318604 25575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:02.318728 25575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:02.326676 25575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:02.326826 25575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:02.327008 25575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:42:02.337600 25575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:42071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=33391
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:02.339002 25575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:02.340442 25575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:02.356232 25624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:02.356263 25626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:02.357522 25629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:02.357532 25575 server_base.cc:1034] running on GCE node
I20260810 23:42:02.358335 25575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:02.359413 25575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:02.360682 25575 hybrid_clock.cc:648] HybridClock initialized: now 1786405322360618 us; error 64 us; skew 500 ppm
I20260810 23:42:02.361018 25575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:02 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405322, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:02.364751 25575 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:42:02.366839 25575 webserver.cc:460] Webserver started at http://127.30.33.1:33391/ using document root <none> and password file <none>
I20260810 23:42:02.367865 25575 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:02.368052 25575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:02.375954 25575 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:42:02.379776 25649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:02.381748 25575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:42:02.381958 25575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "89d70d0a2af04d79b0d1b7f9d943f318"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "565b9f636ffb1ee75c1746a294c33f9f"
server_key_iv: "7ae9aea8b6b3355fa50afaf8142a86b0"
server_key_version: "encryptionkey@0"
I20260810 23:42:02.383402 25575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:02.401049 25575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:02.406042 25575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:02.406308 25575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:02.407577 25575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:02.409227 25575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:02.409338 25575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:02.409430 25575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:02.409483 25575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:02.442499 25575 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:42071
I20260810 23:42:02.442576 25864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:42071 every 8 connection(s)
I20260810 23:42:02.444204 25575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:42:02.454284 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 25575
I20260810 23:42:02.454903 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:42531
--local_ip_for_outbound_sockets=127.30.33.2
--tserver_master_addrs=127.30.33.62:41349
--webserver_port=35229
--webserver_interface=127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:42:02 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405322, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:02.471953 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:02.447798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:54743)
0810 23:42:02.448733 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:02.448788 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:02.450646 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:02.452257 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:02.452265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:02.452860 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:02.453468 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:02.453489 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.455416 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.455421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:02.457441 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:02.457462 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.458723 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.458731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:02.459019 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:02.459866 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:02.459898 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:02.462820 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:02.466762 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:02.466773 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:02.466789 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:02.467376 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:02.467964 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:02.467970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:02.467974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:02.468222 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:02.469103 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:02.469113 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:02.469495 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 23:42:02.469766 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 23:42:02.470462 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":368,"threads_started":1}
I20260810 23:42:02.471953 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:02.447572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:54743 (local address 127.30.33.62:41349)
0810 23:42:02.448211 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:02.448234 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:02.448950 (+ 716us) server_negotiation.cc:407] Connection header received
0810 23:42:02.451232 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:02.451247 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:02.451703 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:02.452072 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:02.453769 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.455212 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.457656 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.458435 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.458606 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:02.463183 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:02.463195 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:02.463214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:02.463262 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:02.466521 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:02.467588 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:02.467597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:02.467604 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:02.467823 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:02.468360 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:02.468364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:02.468366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:02.468930 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:02.469129 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:02.470061 (+ 932us) server_negotiation.cc:299] Negotiation successful
0810 23:42:02.470525 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":264,"threads_started":1}
I20260810 23:42:02.475580 25865 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41349
I20260810 23:42:02.475944 25865 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:02.476758 25865 heartbeater.cc:511] Master 127.30.33.62:41349 requested a full tablet report, sending...
I20260810 23:42:02.479162 25508 ts_manager.cc:194] Registered new tserver with Master: 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071)
I20260810 23:42:02.480991 25508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:54743
W20260810 23:42:02.819343 25875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:02.819746 25875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:02.819901 25875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:02.827632 25875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:02.827809 25875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:02.827986 25875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:42:02.838313 25875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:42531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=35229
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:02.839740 25875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:02.841086 25875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:02.854525 25882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:02.854561 25884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:02.854609 25887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:02.855801 25875 server_base.cc:1034] running on GCE node
I20260810 23:42:02.856575 25875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:02.857671 25875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:02.858915 25875 hybrid_clock.cc:648] HybridClock initialized: now 1786405322858861 us; error 44 us; skew 500 ppm
I20260810 23:42:02.859324 25875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:02 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405322, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:02.862968 25875 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:42:02.864675 25875 webserver.cc:460] Webserver started at http://127.30.33.2:35229/ using document root <none> and password file <none>
I20260810 23:42:02.865442 25875 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:02.865587 25875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:02.871999 25875 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:42:02.875013 25897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:02.876646 25875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:42:02.876822 25875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "1c5b9ebf2e384974b8ba060d72a49696"
format_stamp: "Formatted at 2026-08-10 23:41:59 on dist-test-slave-pww0"
server_key: "c719951ad3a0d446e3271588e5f1d296"
server_key_iv: "d3ccc637868e3a7eb544498e71483770"
server_key_version: "encryptionkey@0"
I20260810 23:42:02.877866 25875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:02.892375 25875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:02.897050 25875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:02.897347 25875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:02.898453 25875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:02.900008 25875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:02.900107 25875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:02.900213 25875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:02.900287 25875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:02.932557 25875 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:42531
I20260810 23:42:02.932619 26099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:42531 every 8 connection(s)
I20260810 23:42:02.934054 25875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:42:02.940191 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 25875
I20260810 23:42:02.940830 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:41719
--local_ip_for_outbound_sockets=127.30.33.3
--tserver_master_addrs=127.30.33.62:41349
--webserver_port=42975
--webserver_interface=127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:42:02 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405322, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:02.956591 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:02.936854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:52433 (local address 127.30.33.62:41349)
0810 23:42:02.937080 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:02.937086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:02.938452 (+ 1366us) server_negotiation.cc:407] Connection header received
0810 23:42:02.940937 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:02.940944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:02.941057 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:02.941170 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:02.942821 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.943795 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.945430 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.946037 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.946173 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:02.950083 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:02.950091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:02.950095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:02.950141 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:02.953157 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:02.953999 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:02.954005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:02.954008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:02.954159 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:02.954639 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:02.954645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:02.954647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:02.954947 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:02.955117 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:02.955941 (+ 824us) server_negotiation.cc:299] Negotiation successful
0810 23:42:02.956206 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:42:02.957654 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:02.937396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:52433)
0810 23:42:02.938256 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:02.938312 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:02.940638 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:02.941437 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:02.941447 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:02.941994 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:02.942619 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:02.942640 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.943968 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.943972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:02.945250 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:02.945263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:02.946303 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:02.946309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:02.946539 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:02.947273 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:02.947299 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:02.949825 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:02.953336 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:02.953347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:02.953364 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:02.953862 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:02.954283 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:02.954289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:02.954292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:02.954517 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:02.955107 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:02.955116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:02.955537 (+ 421us) client_negotiation.cc:769] Sending connection context
0810 23:42:02.955825 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 23:42:02.956436 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":307,"threads_started":1}
I20260810 23:42:02.959498 26101 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41349
I20260810 23:42:02.959892 26101 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:02.960850 26101 heartbeater.cc:511] Master 127.30.33.62:41349 requested a full tablet report, sending...
I20260810 23:42:02.962824 25508 ts_manager.cc:194] Registered new tserver with Master: 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531)
I20260810 23:42:02.964000 25508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:52433
W20260810 23:42:03.302269 26106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:03.302582 26106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:03.302680 26106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:03.310161 26106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:03.310312 26106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:03.310437 26106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:42:03.320631 26106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:41719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=42975
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:03.322003 26106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:03.323345 26106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:03.338001 26118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:03.338824 26123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:03.340379 26119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:03.340756 26106 server_base.cc:1034] running on GCE node
I20260810 23:42:03.341616 26106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:03.342754 26106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:03.343997 26106 hybrid_clock.cc:648] HybridClock initialized: now 1786405323343941 us; error 57 us; skew 500 ppm
I20260810 23:42:03.344349 26106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:03 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405323, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:03.347815 26106 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:42:03.349411 26106 webserver.cc:460] Webserver started at http://127.30.33.3:42975/ using document root <none> and password file <none>
I20260810 23:42:03.350149 26106 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:03.350299 26106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:03.356175 26106 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:42:03.359114 26141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:03.360729 26106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:42:03.360914 26106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "497cfb3944fc4afab27988eabe2836bc"
format_stamp: "Formatted at 2026-08-10 23:42:00 on dist-test-slave-pww0"
server_key: "d76260f5de69dfc08ba98a306d3242a1"
server_key_iv: "47a542198c767a37a75c2dbc64921763"
server_key_version: "encryptionkey@0"
I20260810 23:42:03.361970 26106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:03.380987 26106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:03.386015 26106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:03.386299 26106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:03.387368 26106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:03.388939 26106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:03.389031 26106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:03.389114 26106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:03.389171 26106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:03.421571 26106 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:41719
I20260810 23:42:03.421657 26369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:41719 every 8 connection(s)
I20260810 23:42:03.423142 26106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:42:03.424847 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 26106
Aug 10 23:42:03 dist-test-slave-pww0 krb5kdc[24400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405323, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:03.442315 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.425998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.30.33.62:41349)
0810 23:42:03.426228 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.426235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.426881 (+ 646us) server_negotiation.cc:407] Connection header received
0810 23:42:03.427084 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.427090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.427190 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.427338 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:03.429500 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.430276 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.432310 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.432775 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.432931 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.436309 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:03.436317 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:03.436321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:03.436370 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:03.439301 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:03.440115 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:03.440120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:03.440122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:03.440260 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:03.440642 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:03.440645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:03.440648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:03.440973 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:03.441259 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.441550 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.441749 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:42:03.447736 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.426248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:48771 (local address 127.30.33.62:41349)
0810 23:42:03.429658 (+ 3410us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.429664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.429685 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:42:03.430538 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.430542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.430606 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.430687 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:03.432904 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.433740 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.436341 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.437062 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.437191 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.440981 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:03.440987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:03.440992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:03.441024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:03.444389 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:03.445361 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:03.445365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:03.445367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:03.445482 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:03.446000 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:03.446002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:03.446004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:03.446273 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:03.446379 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.447202 (+ 823us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.447380 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3278,"thread_start_us":935,"threads_started":1}
I20260810 23:42:03.449182 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.426898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:48771)
0810 23:42:03.427918 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:03.427980 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:03.430212 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:03.430993 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:03.431005 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:03.431867 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:03.432694 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.432715 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.433929 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.433936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.436198 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.436217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.437335 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.437344 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.437638 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.438272 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:03.438306 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:03.440717 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:03.444550 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:03.444559 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:03.444577 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:03.445221 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:03.445595 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:03.445600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:03.445603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:03.445897 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:03.446383 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:03.446391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:03.446855 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 23:42:03.447147 (+ 292us) client_negotiation.cc:240] Negotiation successful
0810 23:42:03.447790 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":438,"threads_started":1}
I20260810 23:42:03.450657 26372 heartbeater.cc:347] Connected to a master server at 127.30.33.62:41349
I20260810 23:42:03.450960 26372 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:03.451793 26372 heartbeater.cc:511] Master 127.30.33.62:41349 requested a full tablet report, sending...
I20260810 23:42:03.453285 25508 ts_manager.cc:194] Registered new tserver with Master: 497cfb3944fc4afab27988eabe2836bc (127.30.33.3:41719)
I20260810 23:42:03.454123 25508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:48771
I20260810 23:42:03.457584 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:42:03.464629 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.458551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.30.33.62:41349)
0810 23:42:03.458764 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.458771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.459221 (+ 450us) server_negotiation.cc:407] Connection header received
0810 23:42:03.459451 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.459483 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.459652 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.459792 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:42:03.460179 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.460888 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.462431 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.462882 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.462976 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.463481 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:42:03.464090 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:42:03.464201 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.464337 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.464414 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:42:03.465595 25508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:42:03.473084 25508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:42:03.484843 25865 heartbeater.cc:503] Master 127.30.33.62:41349 was elected leader, sending a full tablet report...
I20260810 23:42:03.496811 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.485477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38868)
0810 23:42:03.486443 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:03.486486 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:03.486671 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:03.487631 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:03.487639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:03.487672 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:03.488255 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.488273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.490545 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.490551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.493221 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.493242 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.495739 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.495746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.495956 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.496015 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 23:42:03.496160 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 23:42:03.496393 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":310,"threads_started":1}
I20260810 23:42:03.497877 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.485679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.30.33.1:42071)
0810 23:42:03.486658 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.486666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.486701 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:42:03.486893 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.486899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.487093 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.487282 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:03.488578 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.490355 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.493450 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.495142 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.495270 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.495570 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.497261 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.497498 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":389,"threads_started":1}
I20260810 23:42:03.502784 25744 tablet_service.cc:1467] Processing CreateTablet for tablet bf660fe92b4749bb97ebb70424e63cf9 (DEFAULT_TABLE table=test-table [id=9f0303bb5b0e4d56b75e3a2a671c1cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:03.505072 25744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf660fe92b4749bb97ebb70424e63cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:03.519085 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.488948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.30.33.2:42531)
0810 23:42:03.496657 (+ 7709us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.496668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.510741 (+ 14073us) server_negotiation.cc:407] Connection header received
0810 23:42:03.511082 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.511091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.511387 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.511601 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:03.513049 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.514596 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.517024 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.517986 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.518124 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.518350 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.518593 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.518768 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7491,"thread_start_us":7337,"threads_started":1}
I20260810 23:42:03.519263 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.487995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:41719 (local address 127.0.0.1:42940)
0810 23:42:03.498822 (+ 10827us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:03.498845 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:03.498980 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:03.511891 (+ 12911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:03.511896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:03.511925 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:03.512129 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.512142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.514614 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.514619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.516669 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.516688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.518451 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.518458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.518662 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.518725 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 23:42:03.518794 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:42:03.518946 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10692,"thread_start_us":24457,"threads_started":1}
I20260810 23:42:03.519066 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.488356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42531 (local address 127.0.0.1:46308)
0810 23:42:03.510504 (+ 22148us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:03.510601 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:03.510880 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:03.512328 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:03.512332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:03.512357 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:03.512851 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.512866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.514775 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.514780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.516873 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.516884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.518218 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.518225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.518403 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.518440 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:42:03.518526 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:42:03.518671 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21987,"thread_start_us":21807,"threads_started":1}
I20260810 23:42:03.520023 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.488117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.30.33.3:41719)
0810 23:42:03.511087 (+ 22970us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.511097 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.511141 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:42:03.511276 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.511284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.511589 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.511808 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:03.512774 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.514426 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.516846 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.518229 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.519167 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.519405 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.519518 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.519684 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22746,"thread_start_us":23932,"threads_started":1}
I20260810 23:42:03.523964 25991 tablet_service.cc:1467] Processing CreateTablet for tablet bf660fe92b4749bb97ebb70424e63cf9 (DEFAULT_TABLE table=test-table [id=9f0303bb5b0e4d56b75e3a2a671c1cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:03.523964 26243 tablet_service.cc:1467] Processing CreateTablet for tablet bf660fe92b4749bb97ebb70424e63cf9 (DEFAULT_TABLE table=test-table [id=9f0303bb5b0e4d56b75e3a2a671c1cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:03.525100 26402 tablet_bootstrap.cc:492] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Bootstrap starting.
I20260810 23:42:03.525640 25991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf660fe92b4749bb97ebb70424e63cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:03.525648 26243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf660fe92b4749bb97ebb70424e63cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:03.528666 26402 tablet_bootstrap.cc:654] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:03.529800 26402 log.cc:824] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:03.533376 26402 tablet_bootstrap.cc:492] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: No bootstrap required, opened a new log
I20260810 23:42:03.533667 26402 ts_tablet_manager.cc:1397] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:42:03.540850 26402 raft_consensus.cc:348] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.541251 26402 raft_consensus.cc:374] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:03.541368 26402 raft_consensus.cc:729] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d70d0a2af04d79b0d1b7f9d943f318, State: Initialized, Role: FOLLOWER
I20260810 23:42:03.542204 26402 consensus_queue.cc:260] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.544736 26404 tablet_bootstrap.cc:492] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Bootstrap starting.
I20260810 23:42:03.545648 26405 tablet_bootstrap.cc:492] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Bootstrap starting.
I20260810 23:42:03.549926 26405 tablet_bootstrap.cc:654] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:03.550508 26404 tablet_bootstrap.cc:654] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:03.551083 26402 ts_tablet_manager.cc:1428] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260810 23:42:03.551528 26405 log.cc:824] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:03.551883 26404 log.cc:824] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:03.554919 26405 tablet_bootstrap.cc:492] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: No bootstrap required, opened a new log
I20260810 23:42:03.555054 26404 tablet_bootstrap.cc:492] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: No bootstrap required, opened a new log
I20260810 23:42:03.555249 26405 ts_tablet_manager.cc:1397] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:42:03.555294 26404 ts_tablet_manager.cc:1397] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 23:42:03.563129 26404 raft_consensus.cc:348] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.563565 26404 raft_consensus.cc:374] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:03.563373 26405 raft_consensus.cc:348] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.563673 26404 raft_consensus.cc:729] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497cfb3944fc4afab27988eabe2836bc, State: Initialized, Role: FOLLOWER
I20260810 23:42:03.563835 26405 raft_consensus.cc:374] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:03.563933 26405 raft_consensus.cc:729] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5b9ebf2e384974b8ba060d72a49696, State: Initialized, Role: FOLLOWER
I20260810 23:42:03.564343 26404 consensus_queue.cc:260] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.564534 26405 consensus_queue.cc:260] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.565589 26372 heartbeater.cc:503] Master 127.30.33.62:41349 was elected leader, sending a full tablet report...
I20260810 23:42:03.565668 26101 heartbeater.cc:503] Master 127.30.33.62:41349 was elected leader, sending a full tablet report...
I20260810 23:42:03.566188 26404 ts_tablet_manager.cc:1428] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260810 23:42:03.567752 26405 ts_tablet_manager.cc:1428] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260810 23:42:03.676110 26373 tablet.cc:2369] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:42:03.686342 26103 tablet.cc:2369] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:42:03.696643 25871 tablet.cc:2369] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:42:03.713806 26410 raft_consensus.cc:482] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:42:03.714124 26410 raft_consensus.cc:504] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.715883 26410 leader_election.cc:288] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071), 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531)
I20260810 23:42:03.722131 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.716177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:42531 (local address 127.30.33.3:47417)
0810 23:42:03.716499 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:03.716519 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:03.716657 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:03.717083 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:03.717088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:03.717119 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:03.717360 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.717372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.718544 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.718549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.720088 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.720101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.721397 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.721403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.721588 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.721624 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:42:03.721729 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:42:03.721874 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":3,"spinlock_wait_cycles":37632}
I20260810 23:42:03.722154 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.716510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:47417 (local address 127.30.33.2:42531)
0810 23:42:03.716712 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.716730 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.716756 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:42:03.716838 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.716844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.716926 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.717035 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:03.717524 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.718413 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.720254 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.721236 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.721342 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.721419 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.721759 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.721890 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:42:03.722604 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.716477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:49599 (local address 127.30.33.1:42071)
0810 23:42:03.716764 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.716769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.717128 (+ 359us) server_negotiation.cc:407] Connection header received
0810 23:42:03.717302 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.717308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.717384 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.717472 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:03.718014 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.718899 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.720795 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.721668 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.721800 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.721901 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.722242 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.722378 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260810 23:42:03.722605 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.716281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.30.33.3:49599)
0810 23:42:03.717019 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:03.717036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:03.717161 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:03.717504 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:03.717507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:03.717532 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:03.717880 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.717891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.719056 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.719059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.720628 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:03.720643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.721880 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.721886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:03.722064 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.722096 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:42:03.722196 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:42:03.722327 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":485,"threads_started":1}
I20260810 23:42:03.723598 26024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf660fe92b4749bb97ebb70424e63cf9" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5b9ebf2e384974b8ba060d72a49696" is_pre_election: true
I20260810 23:42:03.723641 25782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf660fe92b4749bb97ebb70424e63cf9" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" is_pre_election: true
I20260810 23:42:03.724119 25782 raft_consensus.cc:2454] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 0.
I20260810 23:42:03.724119 26024 raft_consensus.cc:2454] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 0.
I20260810 23:42:03.724818 26145 leader_election.cc:302] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 497cfb3944fc4afab27988eabe2836bc, 89d70d0a2af04d79b0d1b7f9d943f318; no voters:
I20260810 23:42:03.725193 26410 raft_consensus.cc:2781] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:42:03.725363 26410 raft_consensus.cc:482] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:42:03.725468 26410 raft_consensus.cc:3037] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:03.727774 26410 raft_consensus.cc:504] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.728401 26410 leader_election.cc:288] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 election: Requested vote from peers 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071), 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531)
I20260810 23:42:03.728962 25782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf660fe92b4749bb97ebb70424e63cf9" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d70d0a2af04d79b0d1b7f9d943f318"
I20260810 23:42:03.729197 25782 raft_consensus.cc:3037] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:03.729219 26024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf660fe92b4749bb97ebb70424e63cf9" candidate_uuid: "497cfb3944fc4afab27988eabe2836bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5b9ebf2e384974b8ba060d72a49696"
I20260810 23:42:03.729442 26024 raft_consensus.cc:3037] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:03.732486 25782 raft_consensus.cc:2454] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 1.
I20260810 23:42:03.732534 26024 raft_consensus.cc:2454] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 497cfb3944fc4afab27988eabe2836bc in term 1.
I20260810 23:42:03.732975 26145 leader_election.cc:302] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 497cfb3944fc4afab27988eabe2836bc, 89d70d0a2af04d79b0d1b7f9d943f318; no voters:
I20260810 23:42:03.733361 26410 raft_consensus.cc:2781] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:42:03.733892 26410 raft_consensus.cc:686] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 1 LEADER]: Becoming Leader. State: Replica: 497cfb3944fc4afab27988eabe2836bc, State: Running, Role: LEADER
I20260810 23:42:03.734404 26410 consensus_queue.cc:237] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } }
I20260810 23:42:03.739192 25507 catalog_manager.cc:5495] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 497cfb3944fc4afab27988eabe2836bc (127.30.33.3). New cstate: current_term: 1 leader_uuid: "497cfb3944fc4afab27988eabe2836bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "497cfb3944fc4afab27988eabe2836bc" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 41719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 } health_report { overall_health: UNKNOWN } } }
I20260810 23:42:03.804368 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.798777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.30.33.3:41719)
0810 23:42:03.798991 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.798997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.799018 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:42:03.799148 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.799152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.799237 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.799381 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:42:03.799836 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.800552 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.802037 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.802454 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.802573 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.803164 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:42:03.803730 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:42:03.803872 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.803984 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.804074 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:42:03.812822 26024 raft_consensus.cc:1261] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Refusing update from remote peer 497cfb3944fc4afab27988eabe2836bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:03.812830 25782 raft_consensus.cc:1261] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Refusing update from remote peer 497cfb3944fc4afab27988eabe2836bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:03.813874 26416 consensus_queue.cc:1035] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5b9ebf2e384974b8ba060d72a49696" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 42531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:03.814221 26410 consensus_queue.cc:1035] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d70d0a2af04d79b0d1b7f9d943f318" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:03.827639 26427 mvcc.cc:205] Tried to move back new op lower bound from 7317116206327066624 to 7317116206019862528. Current Snapshot: MvccSnapshot[applied={T|T < 7317116206327066624}]
I20260810 23:42:03.828076 26424 mvcc.cc:205] Tried to move back new op lower bound from 7317116206327066624 to 7317116206019862528. Current Snapshot: MvccSnapshot[applied={T|T < 7317116206327066624}]
I20260810 23:42:03.845935 25507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:03.846252 25507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:03.850117 25507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Sending DeleteTablet for 3 replicas of tablet bf660fe92b4749bb97ebb70424e63cf9
I20260810 23:42:03.851598 26243 tablet_service.cc:1514] Processing DeleteTablet for tablet bf660fe92b4749bb97ebb70424e63cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:03 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:42940
I20260810 23:42:03.851593 25744 tablet_service.cc:1514] Processing DeleteTablet for tablet bf660fe92b4749bb97ebb70424e63cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:03 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:38868
I20260810 23:42:03.851703 25991 tablet_service.cc:1514] Processing DeleteTablet for tablet bf660fe92b4749bb97ebb70424e63cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:03 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:46308
I20260810 23:42:03.853247 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 25575
I20260810 23:42:03.864332 26446 tablet_replica.cc:330] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: stopping tablet replica
I20260810 23:42:03.865129 26446 raft_consensus.cc:2229] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:03.865938 26446 raft_consensus.cc:2258] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:03.868849 26446 ts_tablet_manager.cc:1905] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:03.869612 26443 tablet_replica.cc:330] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: stopping tablet replica
I20260810 23:42:03.870182 26443 raft_consensus.cc:2229] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:42:03.871076 26443 raft_consensus.cc:2258] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:03.873677 26443 ts_tablet_manager.cc:1905] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:03.882117 26443 ts_tablet_manager.cc:1918] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:03.882422 26445 tablet_replica.cc:330] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: stopping tablet replica
I20260810 23:42:03.882469 26443 log.cc:1196] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/bf660fe92b4749bb97ebb70424e63cf9
I20260810 23:42:03.883128 26445 raft_consensus.cc:2229] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:03.883245 26443 ts_tablet_manager.cc:1939] T bf660fe92b4749bb97ebb70424e63cf9 P 497cfb3944fc4afab27988eabe2836bc: Deleting consensus metadata
I20260810 23:42:03.883848 26445 raft_consensus.cc:2258] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:03.887429 26446 ts_tablet_manager.cc:1918] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:03.888052 26446 log.cc:1196] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/bf660fe92b4749bb97ebb70424e63cf9
I20260810 23:42:03.888094 25476 catalog_manager.cc:4841] TS 497cfb3944fc4afab27988eabe2836bc (127.30.33.3:41719): tablet bf660fe92b4749bb97ebb70424e63cf9 (table test-table [id=9f0303bb5b0e4d56b75e3a2a671c1cc7]) successfully deleted
I20260810 23:42:03.888805 26446 ts_tablet_manager.cc:1939] T bf660fe92b4749bb97ebb70424e63cf9 P 1c5b9ebf2e384974b8ba060d72a49696: Deleting consensus metadata
I20260810 23:42:03.889128 26445 ts_tablet_manager.cc:1905] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:03.890847 25472 catalog_manager.cc:4841] TS 1c5b9ebf2e384974b8ba060d72a49696 (127.30.33.2:42531): tablet bf660fe92b4749bb97ebb70424e63cf9 (table test-table [id=9f0303bb5b0e4d56b75e3a2a671c1cc7]) successfully deleted
I20260810 23:42:03.897446 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:03.854122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.30.33.1:42071)
0810 23:42:03.854754 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:03.854761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:03.854782 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:42:03.854852 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:03.854858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:03.854977 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:03.855076 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:03.857444 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.858727 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.869456 (+ 10729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:03.872446 (+ 2990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:03.872645 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:03.877062 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:03.877084 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:03.877101 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:03.877153 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:03.893165 (+ 16012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:03.894419 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:03.894430 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:03.894434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:03.894651 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:03.895216 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:03.895223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:03.895226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:03.895824 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:03.896163 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:03.896786 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 23:42:03.897055 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469}
I20260810 23:42:03.897604 26445 ts_tablet_manager.cc:1918] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:03.897832 26445 log.cc:1196] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/bf660fe92b4749bb97ebb70424e63cf9
I20260810 23:42:03.898172 25859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:04.029021 26445 ts_tablet_manager.cc:1939] T bf660fe92b4749bb97ebb70424e63cf9 P 89d70d0a2af04d79b0d1b7f9d943f318: Deleting consensus metadata
I20260810 23:42:04.029659 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 25575
W20260810 23:42:04.054652 25474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.33.1:42071 (error 108)
I20260810 23:42:04.055032 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 25875
I20260810 23:42:04.057103 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.056226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38884)
0810 23:42:04.056714 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.056853 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":259,"threads_started":1}
W20260810 23:42:04.057425 25474 catalog_manager.cc:4584] TS 89d70d0a2af04d79b0d1b7f9d943f318 (127.30.33.1:42071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf660fe92b4749bb97ebb70424e63cf9: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
I20260810 23:42:04.073148 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.055929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.30.33.2:42531)
0810 23:42:04.056167 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:04.056174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:04.056198 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:42:04.056393 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:04.056399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:04.056508 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:04.056622 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:04.058503 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:04.059830 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:04.061440 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:04.062094 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:04.062244 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:04.065129 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:04.065154 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:04.065177 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:04.065269 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:04.069535 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:04.070476 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:04.070487 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:04.070492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:04.070679 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:04.071195 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:04.071201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:04.071204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:04.071816 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:04.072157 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:04.072484 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 23:42:04.072754 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 23:42:04.073927 26092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:04.111616 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.110945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38886)
0810 23:42:04.111254 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.111386 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:42:04.112965 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.112459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38892)
0810 23:42:04.112641 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.112772 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:42:04.165946 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 25875
I20260810 23:42:04.182274 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.181674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38894)
0810 23:42:04.181910 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.182036 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:42:04.183629 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.183095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38900)
0810 23:42:04.183275 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.183377 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:42:04.194988 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 26106
I20260810 23:42:04.209877 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.195984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.30.33.3:41719)
0810 23:42:04.196240 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:04.196247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:04.196268 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:42:04.196385 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:04.196389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:04.196466 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:04.196554 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:04.197912 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:04.198730 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:04.200176 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:04.200640 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:04.200765 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:04.203425 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:04.203444 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:04.203524 (+ 80us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:04.203577 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:04.206722 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:04.207573 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:04.207581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:04.207583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:04.207766 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:04.208191 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:04.208195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:04.208197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:04.208598 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:04.208939 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:04.209233 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 23:42:04.209479 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260810 23:42:04.210667 26360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:04.264336 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.263804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38906)
0810 23:42:04.264047 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.264153 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 23:42:04.265607 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.265136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38916)
0810 23:42:04.265369 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.265461 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260810 23:42:04.336953 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 26106
I20260810 23:42:04.360198 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 25437
I20260810 23:42:04.361487 25565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:04.476867 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:04.476371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:42071 (local address 127.0.0.1:38930)
0810 23:42:04.476567 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:04.476679 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:42:04.477246 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 25437
I20260810 23:42:04.508344 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41349
--webserver_interface=127.30.33.62
--webserver_port=38839
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:04.857811 26456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:04.858150 26456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:04.858248 26456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:04.865679 26456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:42:04.865815 26456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:04.865863 26456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:04.865921 26456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:42:04.865962 26456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:42:04.876202 26456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:04.877562 26456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:04.878969 26456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:04.891119 26469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:04.891101 26474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:04.891027 26478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:04.892058 26456 server_base.cc:1034] running on GCE node
I20260810 23:42:04.892899 26456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:04.894374 26456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:04.895541 26456 hybrid_clock.cc:648] HybridClock initialized: now 1786405324895488 us; error 52 us; skew 500 ppm
I20260810 23:42:04.895902 26456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:04 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405324, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:04.899027 26456 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:42:04.900671 26456 webserver.cc:460] Webserver started at http://127.30.33.62:38839/ using document root <none> and password file <none>
I20260810 23:42:04.901366 26456 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:04.901472 26456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:04.907066 26456 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 23:42:04.910123 26492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:04.911669 26456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:42:04.911839 26456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "ee93b0ea15964746b0cc39308d6893ac"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "4ef776f85893f212487248342f9ae21f"
server_key_iv: "60c2b311a78d8c3c2d898d531b91cbb3"
server_key_version: "encryptionkey@0"
I20260810 23:42:04.912861 26456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:04.939278 26456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:04.944046 26456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:04.944337 26456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:04.963577 26456 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:41349
I20260810 23:42:04.963591 26582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:41349 every 8 connection(s)
I20260810 23:42:04.965247 26456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:42:04.969611 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 26456
I20260810 23:42:04.970528 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:42071
--local_ip_for_outbound_sockets=127.30.33.1
--tserver_master_addrs=127.30.33.62:41349
--webserver_port=33391
--webserver_interface=127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:42:04.972677 26585 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:42:04.976276 26585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Bootstrap starting.
I20260810 23:42:04.997712 26585 log.cc:824] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:05.014899 26585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:42:05.015499 26585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac: Bootstrap complete.
I20260810 23:42:05.025110 26585 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:05.026149 26585 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee93b0ea15964746b0cc39308d6893ac, State: Initialized, Role: FOLLOWER
I20260810 23:42:05.026899 26585 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:05.027176 26585 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:42:05.027311 26585 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:42:05.027516 26585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:42:05.031692 26585 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:05.032253 26585 leader_election.cc:302] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee93b0ea15964746b0cc39308d6893ac; no voters:
I20260810 23:42:05.032943 26585 leader_election.cc:288] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:42:05.033103 26592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:42:05.034291 26592 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [term 3 LEADER]: Becoming Leader. State: Replica: ee93b0ea15964746b0cc39308d6893ac, State: Running, Role: LEADER
I20260810 23:42:05.034858 26592 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } }
I20260810 23:42:05.035279 26585 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:42:05.037648 26594 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ee93b0ea15964746b0cc39308d6893ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } } }
I20260810 23:42:05.037626 26595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee93b0ea15964746b0cc39308d6893ac. Latest consensus state: current_term: 3 leader_uuid: "ee93b0ea15964746b0cc39308d6893ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee93b0ea15964746b0cc39308d6893ac" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41349 } } }
I20260810 23:42:05.038065 26594 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:05.038064 26595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee93b0ea15964746b0cc39308d6893ac [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:05.040694 26599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:42:05.050026 26599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:42:05.051949 26599 catalog_manager.cc:1259] Loaded cluster ID: 9af2e1d7d5094432a86aa32a38dc991e
I20260810 23:42:05.052078 26599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 23:42:05.054459 26599 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1de881f9ad google::LogMessage::Fail() at ??:0
@ 0x7f1de882288a google::LogMessage::SendToLog() at ??:0
@ 0x7f1de881f4de google::LogMessage::Flush() at ??:0
@ 0x7f1de8820a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1dfe256512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1dfe253976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1dfe2db610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1dfe2db439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1deb46b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1deb46e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1deb46e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1deb449eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1dee7236db start_thread at ??:0
@ 0x7f1de76b961f clone at ??:0
W20260810 23:42:05.374538 26588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:05.374867 26588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:05.374969 26588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:05.382827 26588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:05.382972 26588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:05.383101 26588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:42:05.393664 26588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:42071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=33391
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:05.395093 26588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:05.396536 26588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:05.410943 26664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:05.410972 26665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:05.411048 26667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:05.411582 26588 server_base.cc:1034] running on GCE node
I20260810 23:42:05.412304 26588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:05.413375 26588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:05.414623 26588 hybrid_clock.cc:648] HybridClock initialized: now 1786405325414556 us; error 54 us; skew 500 ppm
I20260810 23:42:05.414983 26588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:05 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405325, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:05.418423 26588 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:42:05.420290 26588 webserver.cc:460] Webserver started at http://127.30.33.1:33391/ using document root <none> and password file <none>
I20260810 23:42:05.421047 26588 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:05.421183 26588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:05.427309 26588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:42:05.430438 26684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:05.431957 26588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:42:05.432137 26588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "89d70d0a2af04d79b0d1b7f9d943f318"
format_stamp: "Formatted at 2026-08-10 23:41:58 on dist-test-slave-pww0"
server_key: "565b9f636ffb1ee75c1746a294c33f9f"
server_key_iv: "7ae9aea8b6b3355fa50afaf8142a86b0"
server_key_version: "encryptionkey@0"
I20260810 23:42:05.433176 26588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:05.446295 26588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:05.450812 26588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:05.451097 26588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:05.452181 26588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:05.453660 26588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:05.453773 26588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:05.454005 26588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:05.454128 26588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:05.487530 26588 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:42071
I20260810 23:42:05.487628 26892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:42071 every 8 connection(s)
I20260810 23:42:05.489058 26588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:42:05.494215 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:05.492228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:47883)
0810 23:42:05.493028 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:05.493240 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":306,"threads_started":1}
W20260810 23:42:05.494990 26895 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:41349 (0 consecutive failures): Network error: Failed to ping master at 127.30.33.62:41349: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
I20260810 23:42:05.496037 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:05.495627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:50105)
0810 23:42:05.495809 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:05.495895 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:42:05.496656 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 26588
I20260810 23:42:05.497135 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:05.496703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:46597)
0810 23:42:05.496879 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:05.496977 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:42:05.497393 26895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:42:05.497491 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:42531
--local_ip_for_outbound_sockets=127.30.33.2
--tserver_master_addrs=127.30.33.62:41349
--webserver_port=35229
--webserver_interface=127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:05.850934 26904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:05.851300 26904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:05.851432 26904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:05.859367 26904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:05.859560 26904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:05.859704 26904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:42:05.870036 26904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:42531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=35229
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:05.871440 26904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:05.873178 26904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:05.887272 26918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:05.887351 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:05.887271 26921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:05.887854 26904 server_base.cc:1034] running on GCE node
I20260810 23:42:05.888702 26904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:05.889752 26904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:05.890980 26904 hybrid_clock.cc:648] HybridClock initialized: now 1786405325890924 us; error 48 us; skew 500 ppm
I20260810 23:42:05.891322 26904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:05 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405325, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:05.894575 26904 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:42:05.896250 26904 webserver.cc:460] Webserver started at http://127.30.33.2:35229/ using document root <none> and password file <none>
I20260810 23:42:05.897008 26904 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:05.897143 26904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:05.903090 26904 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:42:05.906466 26941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:05.908300 26904 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 23:42:05.908519 26904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "1c5b9ebf2e384974b8ba060d72a49696"
format_stamp: "Formatted at 2026-08-10 23:41:59 on dist-test-slave-pww0"
server_key: "c719951ad3a0d446e3271588e5f1d296"
server_key_iv: "d3ccc637868e3a7eb544498e71483770"
server_key_version: "encryptionkey@0"
I20260810 23:42:05.909691 26904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:05.929807 26904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:05.934482 26904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:05.934783 26904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:05.935950 26904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:05.937515 26904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:05.937613 26904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:05.937727 26904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:05.937794 26904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:05.970423 26904 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:42531
I20260810 23:42:05.970479 27148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:42531 every 8 connection(s)
I20260810 23:42:05.971953 26904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:42:05.977486 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:05.975414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:40455)
0810 23:42:05.976276 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:05.976490 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":316,"threads_started":1}
W20260810 23:42:05.978226 27149 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:41349 (0 consecutive failures): Network error: Failed to ping master at 127.30.33.62:41349: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
I20260810 23:42:05.979243 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:05.978775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:59205)
0810 23:42:05.978983 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:05.979077 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 23:42:05.979223 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 26904
I20260810 23:42:05.980001 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:41719
--local_ip_for_outbound_sockets=127.30.33.3
--tserver_master_addrs=127.30.33.62:41349
--webserver_port=42975
--webserver_interface=127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:42:05.980396 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:05.979971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:43355)
0810 23:42:05.980141 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:05.980253 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:42:05.980661 27149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:42:06.334357 27157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:06.334666 27157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:06.334789 27157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:06.342537 27157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:06.342677 27157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:06.342828 27157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:42:06.353629 27157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:41719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=42975
--tserver_master_addrs=127.30.33.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:06.355104 27157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:06.356637 27157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:06.370569 27169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:06.370661 27164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:06.370661 27163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:06.371797 27157 server_base.cc:1034] running on GCE node
I20260810 23:42:06.372562 27157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:06.373703 27157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:06.374936 27157 hybrid_clock.cc:648] HybridClock initialized: now 1786405326374876 us; error 51 us; skew 500 ppm
I20260810 23:42:06.375301 27157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:06 dist-test-slave-pww0 krb5kdc[24400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405326, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:06.378819 27157 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:42:06.380617 27157 webserver.cc:460] Webserver started at http://127.30.33.3:42975/ using document root <none> and password file <none>
I20260810 23:42:06.381390 27157 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:06.381577 27157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:06.387904 27157 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:42:06.391139 27190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:06.392863 27157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:42:06.393075 27157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "497cfb3944fc4afab27988eabe2836bc"
format_stamp: "Formatted at 2026-08-10 23:42:00 on dist-test-slave-pww0"
server_key: "d76260f5de69dfc08ba98a306d3242a1"
server_key_iv: "47a542198c767a37a75c2dbc64921763"
server_key_version: "encryptionkey@0"
I20260810 23:42:06.394189 27157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:06.425165 27157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:06.429823 27157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:06.430107 27157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:06.431243 27157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:06.432787 27157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:06.432893 27157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:06.433007 27157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:06.433081 27157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:06.466367 27157 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:41719
I20260810 23:42:06.466449 27404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:41719 every 8 connection(s)
I20260810 23:42:06.467952 27157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:42:06.472011 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 27157
I20260810 23:42:06.473342 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:06.471132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:47019)
0810 23:42:06.471984 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:06.472233 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":403,"threads_started":1}
W20260810 23:42:06.474436 27405 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:41349 (0 consecutive failures): Network error: Failed to ping master at 127.30.33.62:41349: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
I20260810 23:42:06.475389 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:06.475030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:39425)
0810 23:42:06.475178 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:06.475261 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:42:06.476326 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:06.475971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:58935)
0810 23:42:06.476119 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:06.476204 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:42:06.476577 27405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:42:06.498895 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:06.497968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:51777)
0810 23:42:06.498486 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:06.498607 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":254,"threads_started":1}
I20260810 23:42:06.982250 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:06.981335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:50743)
0810 23:42:06.981867 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:06.981995 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260810 23:42:07.478011 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:07.477157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:51715)
0810 23:42:07.477650 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:07.477757 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":237,"threads_started":1}
I20260810 23:42:07.500692 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:07.499919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:54559)
0810 23:42:07.500434 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:07.500517 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":262,"threads_started":1}
I20260810 23:42:07.984164 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:07.983360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:44145)
0810 23:42:07.983894 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:07.983985 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":277,"threads_started":1}
I20260810 23:42:08.479967 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:08.479057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:60975)
0810 23:42:08.479697 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:08.479789 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":308,"threads_started":1}
I20260810 23:42:08.502682 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:08.501694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:43393)
0810 23:42:08.502410 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:08.502496 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":390,"threads_started":1}
I20260810 23:42:08.986135 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:08.985265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:56203)
0810 23:42:08.985840 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:08.985941 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":304,"threads_started":1}
I20260810 23:42:09.482021 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:09.481103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:60413)
0810 23:42:09.481682 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:09.481793 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":310,"threads_started":1}
I20260810 23:42:09.504577 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:09.503732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:58107)
0810 23:42:09.504268 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:09.504364 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260810 23:42:09.987973 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:09.987102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:45593)
0810 23:42:09.987666 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:09.987759 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":275,"threads_started":1}
I20260810 23:42:10.484099 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:10.483209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:34145)
0810 23:42:10.483791 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:10.483890 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":314,"threads_started":1}
I20260810 23:42:10.506474 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:10.505652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:42527)
0810 23:42:10.506173 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:10.506286 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":257,"threads_started":1}
I20260810 23:42:10.989693 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:10.988911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:57017)
0810 23:42:10.989421 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:10.989511 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":236,"threads_started":1}
I20260810 23:42:11.485886 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:11.485091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:53899)
0810 23:42:11.485587 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:11.485680 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":249,"threads_started":1}
I20260810 23:42:11.508350 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:11.507503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:41483)
0810 23:42:11.508043 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:11.508147 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":258,"threads_started":1}
I20260810 23:42:11.991664 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:11.990751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:57755)
0810 23:42:11.991286 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:11.991387 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":282,"threads_started":1}
I20260810 23:42:12.487871 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:12.486884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:50929)
0810 23:42:12.487488 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:12.487653 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":296,"threads_started":1}
I20260810 23:42:12.510284 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:12.509309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:40063)
0810 23:42:12.509914 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:12.510006 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":283,"threads_started":1}
I20260810 23:42:12.993788 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:12.992791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:39593)
0810 23:42:12.993443 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:12.993551 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":304,"threads_started":1}
I20260810 23:42:13.490182 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:13.489143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:45709)
0810 23:42:13.489793 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:13.489896 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":317,"threads_started":1}
I20260810 23:42:13.512332 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:13.511411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:44211)
0810 23:42:13.512043 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:13.512142 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":312,"threads_started":1}
I20260810 23:42:13.995833 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:13.994974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:43449)
0810 23:42:13.995515 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:13.995621 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260810 23:42:14.492012 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:14.491191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:36821)
0810 23:42:14.491750 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:14.491839 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":311,"threads_started":1}
I20260810 23:42:14.514183 27454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:14.513373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:39911)
0810 23:42:14.513853 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:14.513985 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":235,"threads_started":1}
I20260810 23:42:14.997838 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:14.996925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:54245)
0810 23:42:14.997514 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:14.997606 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":301,"threads_started":1}
I20260810 23:42:15.494040 27457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:15.493114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:37657)
0810 23:42:15.493721 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:15.493855 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":311,"threads_started":1}
I20260810 23:42:15.516264 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:15.515301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:44767)
0810 23:42:15.515970 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:15.516078 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":403,"threads_started":1}
I20260810 23:42:15.999853 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:15.998992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:41171)
0810 23:42:15.999542 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:15.999636 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":268,"threads_started":1}
I20260810 23:42:16.496097 27464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:16.495242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:39563)
0810 23:42:16.495815 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:16.495913 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":321,"threads_started":1}
I20260810 23:42:16.518200 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:16.517352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:39203)
0810 23:42:16.517908 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:16.517999 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":284,"threads_started":1}
I20260810 23:42:17.001801 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:17.000940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:38967)
0810 23:42:17.001448 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:17.001551 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260810 23:42:17.498162 27474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:17.497268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:47491)
0810 23:42:17.497850 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:17.497957 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":271,"threads_started":1}
I20260810 23:42:17.520188 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:17.519335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:39697)
0810 23:42:17.519906 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:17.519998 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":304,"threads_started":1}
I20260810 23:42:18.003851 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:18.002931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:52991)
0810 23:42:18.003529 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:18.003639 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
I20260810 23:42:18.500053 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:18.499237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:50337)
0810 23:42:18.499794 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:18.499884 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
I20260810 23:42:18.522002 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:18.521181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:55449)
0810 23:42:18.521711 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:18.521812 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":257,"threads_started":1}
I20260810 23:42:19.005774 27486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:19.004875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:51727)
0810 23:42:19.005451 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:19.005558 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":282,"threads_started":1}
I20260810 23:42:19.516490 27487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:19.501164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:41769)
0810 23:42:19.516106 (+ 14942us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:19.516243 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14779,"threads_started":1}
I20260810 23:42:19.524098 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:19.522969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:41639)
0810 23:42:19.523668 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:19.523795 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":272,"threads_started":1}
I20260810 23:42:20.007820 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:20.006862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:49223)
0810 23:42:20.007431 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:20.007602 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
I20260810 23:42:20.518864 27494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:20.517905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:60677)
0810 23:42:20.518490 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:20.518645 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":293,"threads_started":1}
I20260810 23:42:20.526392 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:20.525473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.1:58371)
0810 23:42:20.526055 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:20.526198 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":287,"threads_started":1}
I20260810 23:42:21.010092 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:21.009198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.2:33801)
0810 23:42:21.009754 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:21.009873 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":277,"threads_started":1}
I20260810 23:42:21.484382 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 26588
I20260810 23:42:21.505316 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:21.485688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.30.33.1:42071)
0810 23:42:21.486568 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:21.486584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:21.486629 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:42:21.488623 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:21.488634 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:21.489134 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:21.489507 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:21.491263 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:21.492619 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:21.493985 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:21.494730 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:21.494925 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:21.498020 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:21.498034 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:21.498059 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:21.498114 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:21.501373 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:21.502254 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:21.502263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:21.502271 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:21.502475 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:21.502924 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:21.502928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:21.502930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:21.503627 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:21.503975 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:21.504524 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 23:42:21.505015 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":553,"threads_started":1}
I20260810 23:42:21.506850 26887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:21.521054 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:21.520189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41349 (local address 127.30.33.3:42951)
0810 23:42:21.520746 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:21.520861 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":300,"threads_started":1}
I20260810 23:42:21.593837 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 26588
I20260810 23:42:21.622606 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 26904
I20260810 23:42:21.642578 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:21.623671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.30.33.2:42531)
0810 23:42:21.624239 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:21.624255 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:21.624294 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:42:21.625975 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:21.625986 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:21.626443 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:21.626823 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:21.628540 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:21.629859 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:21.631671 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:21.632415 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:21.632614 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:21.635584 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:21.635597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:21.635622 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:21.635673 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:21.638736 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:21.639610 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:21.639618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:21.639627 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:21.639847 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:21.640322 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:21.640326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:21.640328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:21.640944 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:21.641273 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:21.641809 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 23:42:21.642278 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260810 23:42:21.644212 27140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:21.732942 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 26904
I20260810 23:42:21.767086 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 27157
I20260810 23:42:21.788064 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:21.768350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.30.33.3:41719)
0810 23:42:21.769048 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:21.769067 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:21.769115 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:42:21.770987 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:21.771000 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:21.771602 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:21.772025 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:21.773898 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:21.775197 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:21.776699 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:21.777456 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:21.777661 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:21.780766 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:21.780775 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:21.780795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:21.780844 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:21.784090 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:21.784921 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:21.784929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:21.784938 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:21.785157 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:21.785617 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:21.785621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:21.785624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:21.786244 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:21.786571 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:21.787104 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 23:42:21.787643 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":345,"threads_started":1}
I20260810 23:42:21.790272 27396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:21.879658 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 27157
2026-08-10T23:42:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25618 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:42:22 dist-test-slave-pww0 krb5kdc[27521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:42:22 dist-test-slave-pww0 krb5kdc[27521](info): set up 2 sockets
Aug 10 23:42:22 dist-test-slave-pww0 krb5kdc[27521](info): commencing operation
krb5kdc: starting...
W20260810 23:42:23.197762 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.005s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:42:23 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:42:23.524021 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36009
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:23.874864 27540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:23.875170 27540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:23.875294 27540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:23.882678 27540 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:42:23.882793 27540 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:42:23.882833 27540 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:42:23.882906 27540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:42:23.882980 27540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:23.883069 27540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:23.883112 27540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:42:23.883184 27540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:42:23.893576 27540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:23.894968 27540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:23.896389 27540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:23.908239 27549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:23.908286 27550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:23.908311 27552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:23.909349 27540 server_base.cc:1034] running on GCE node
I20260810 23:42:23.910162 27540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:23.911682 27540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:23.912917 27540 hybrid_clock.cc:648] HybridClock initialized: now 1786405343912850 us; error 40 us; skew 500 ppm
I20260810 23:42:23.913260 27540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:23 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405343, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:23.916450 27540 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:42:23.918057 27540 webserver.cc:460] Webserver started at http://127.30.33.62:41751/ using document root <none> and password file <none>
I20260810 23:42:23.918793 27540 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:23.918898 27540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:23.919226 27540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:42:23.922299 27540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "9d31291a6aec436db87c30f3fb15f59b"
format_stamp: "Formatted at 2026-08-10 23:42:23 on dist-test-slave-pww0"
server_key: "bc15556cd5023f023deef70e0f3da139"
server_key_iv: "910405615564914e68c9ea54838158be"
server_key_version: "encryptionkey@0"
I20260810 23:42:23.923070 27540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "9d31291a6aec436db87c30f3fb15f59b"
format_stamp: "Formatted at 2026-08-10 23:42:23 on dist-test-slave-pww0"
server_key: "bc15556cd5023f023deef70e0f3da139"
server_key_iv: "910405615564914e68c9ea54838158be"
server_key_version: "encryptionkey@0"
I20260810 23:42:23.928375 27540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 23:42:23.931981 27564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:23.933377 27540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:42:23.933553 27540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "9d31291a6aec436db87c30f3fb15f59b"
format_stamp: "Formatted at 2026-08-10 23:42:23 on dist-test-slave-pww0"
server_key: "bc15556cd5023f023deef70e0f3da139"
server_key_iv: "910405615564914e68c9ea54838158be"
server_key_version: "encryptionkey@0"
I20260810 23:42:23.933773 27540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:23.977713 27540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:23.982372 27540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:23.982666 27540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:24.001993 27540 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:36009
I20260810 23:42:24.002012 27665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:36009 every 8 connection(s)
I20260810 23:42:24.003489 27540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:42:24.007550 27667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:24.008455 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 27540
I20260810 23:42:24.008648 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:42:24.009312 30852 external_mini_cluster.cc:1428] Setting key 963f7f46ff28152817c4dd2425178b13
I20260810 23:42:24.020581 27667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Bootstrap starting.
Aug 10 23:42:24 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:24.024318 27667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:24.025599 27667 log.cc:824] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:24.029233 27667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: No bootstrap required, opened a new log
I20260810 23:42:24.033087 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:24.011514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.30.33.62:36009)
0810 23:42:24.012371 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:24.012384 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:24.012427 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:42:24.014085 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:24.014111 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:24.014549 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:24.014937 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:24.016541 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:24.017911 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:24.019355 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:24.020122 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:24.020335 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:24.024988 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:24.024998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:24.025015 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:24.025058 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:24.028525 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:24.029447 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:24.029454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:24.029461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:24.029647 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:24.030135 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:24.030138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:24.030140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:24.030760 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:24.030994 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:24.031562 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 23:42:24.031967 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":381,"threads_started":1}
I20260810 23:42:24.037981 27667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:24.038378 27667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:24.038496 27667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d31291a6aec436db87c30f3fb15f59b, State: Initialized, Role: FOLLOWER
I20260810 23:42:24.039113 27667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:24.039358 27667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:42:24.039549 27667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:42:24.039718 27667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:24.041985 27667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:24.042513 27667 leader_election.cc:302] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d31291a6aec436db87c30f3fb15f59b; no voters:
I20260810 23:42:24.043123 27667 leader_election.cc:288] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:42:24.043239 27684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:42:24.043958 27684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 LEADER]: Becoming Leader. State: Replica: 9d31291a6aec436db87c30f3fb15f59b, State: Running, Role: LEADER
I20260810 23:42:24.044515 27684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:24.045307 27667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:42:24.047297 27686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d31291a6aec436db87c30f3fb15f59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } } }
I20260810 23:42:24.047271 27687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d31291a6aec436db87c30f3fb15f59b. Latest consensus state: current_term: 1 leader_uuid: "9d31291a6aec436db87c30f3fb15f59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } } }
I20260810 23:42:24.047791 27686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:24.047916 27687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:24.049935 27691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:42:24.055541 27691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:42:24.066385 27691 catalog_manager.cc:1347] Generated new cluster ID: 7add57300a114fc886eecd1763a8ebc3
I20260810 23:42:24.066519 27691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:42:24.085454 27691 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:42:24.086963 27691 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:42:24.107127 27691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Generated new TSK 0
I20260810 23:42:24.108162 27691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:42:24.193673 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36009
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:24.556901 27716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:24.557209 27716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:24.557337 27716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:24.564929 27716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:24.565076 27716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:24.565227 27716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:42:24.575646 27716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:24.577010 27716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:24.578367 27716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:24.591245 27727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:24.591835 27733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:24.592945 27737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:24.593961 27716 server_base.cc:1034] running on GCE node
I20260810 23:42:24.594630 27716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:24.595721 27716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:24.596967 27716 hybrid_clock.cc:648] HybridClock initialized: now 1786405344596905 us; error 56 us; skew 500 ppm
I20260810 23:42:24.597324 27716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:24 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405344, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:24.603699 27716 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:42:24.605370 27716 webserver.cc:460] Webserver started at http://127.30.33.1:46805/ using document root <none> and password file <none>
I20260810 23:42:24.606240 27716 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:24.606357 27716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:24.606698 27716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:42:24.610108 27716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "3a5a2ce46b444e07acd9b9096c47e084"
format_stamp: "Formatted at 2026-08-10 23:42:24 on dist-test-slave-pww0"
server_key: "bdeabde505ed412e2efb187dbdb51143"
server_key_iv: "ab5a1352dd98aa362151170326ccd128"
server_key_version: "encryptionkey@0"
I20260810 23:42:24.611013 27716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "3a5a2ce46b444e07acd9b9096c47e084"
format_stamp: "Formatted at 2026-08-10 23:42:24 on dist-test-slave-pww0"
server_key: "bdeabde505ed412e2efb187dbdb51143"
server_key_iv: "ab5a1352dd98aa362151170326ccd128"
server_key_version: "encryptionkey@0"
I20260810 23:42:24.616808 27716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:42:24.620527 27748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:24.621966 27716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:42:24.622139 27716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "3a5a2ce46b444e07acd9b9096c47e084"
format_stamp: "Formatted at 2026-08-10 23:42:24 on dist-test-slave-pww0"
server_key: "bdeabde505ed412e2efb187dbdb51143"
server_key_iv: "ab5a1352dd98aa362151170326ccd128"
server_key_version: "encryptionkey@0"
I20260810 23:42:24.622356 27716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:24.656494 27716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:24.661823 27716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:24.662092 27716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:24.663102 27716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:24.664569 27716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:24.664647 27716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:24.664742 27716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:24.664794 27716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:24.697899 27716 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:34165
I20260810 23:42:24.697978 27972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:34165 every 8 connection(s)
I20260810 23:42:24.699249 27716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:42:24.708307 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 27716
I20260810 23:42:24.708532 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:42:24.709263 30852 external_mini_cluster.cc:1428] Setting key 97c097cf2fc76b0404d13257979f3b69
Aug 10 23:42:24 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405344, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:24.720767 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:24.701975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:40997 (local address 127.30.33.62:36009)
0810 23:42:24.702431 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:24.702436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:24.703512 (+ 1076us) server_negotiation.cc:407] Connection header received
0810 23:42:24.705576 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:24.705581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:24.705670 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:24.705767 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:24.707511 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:24.708538 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:24.710419 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:24.711025 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:24.711175 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:24.714908 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:24.714914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:24.714918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:24.714954 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:24.717496 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:24.718285 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:24.718290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:24.718292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:24.718423 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:24.718882 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:24.718885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:24.718887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:24.719190 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:24.719331 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:24.720142 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 23:42:24.720392 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260810 23:42:24.721832 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:24.702564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:40997)
0810 23:42:24.703293 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:24.703342 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:24.705282 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:24.706043 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:24.706056 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:24.706620 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:24.707287 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:24.707307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:24.708726 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:24.708732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:24.710197 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:24.710215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:24.711315 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:24.711323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:24.711590 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:24.712280 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:24.712306 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:24.714635 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:24.717662 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:24.717673 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:24.717694 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:24.718150 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:24.718538 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:24.718543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:24.718547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:24.718764 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:24.719324 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:24.719332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:24.719752 (+ 420us) client_negotiation.cc:769] Sending connection context
0810 23:42:24.720020 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 23:42:24.720612 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":302,"threads_started":1}
I20260810 23:42:24.723562 27973 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36009
I20260810 23:42:24.723879 27973 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:24.724637 27973 heartbeater.cc:511] Master 127.30.33.62:36009 requested a full tablet report, sending...
I20260810 23:42:24.727038 27596 ts_manager.cc:194] Registered new tserver with Master: 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165)
I20260810 23:42:24.728668 27596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:40997
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:42:24.778954 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36009
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:25.142740 27982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:25.143057 27982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:25.143160 27982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:25.150929 27982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:25.151117 27982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:25.151278 27982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:42:25.161839 27982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:25.163209 27982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:25.164659 27982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:25.178851 28001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:25.178910 27992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:25.179481 27988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:25.180002 27982 server_base.cc:1034] running on GCE node
I20260810 23:42:25.180708 27982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:25.181746 27982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:25.182960 27982 hybrid_clock.cc:648] HybridClock initialized: now 1786405345182894 us; error 60 us; skew 500 ppm
I20260810 23:42:25.183322 27982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:25 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405345, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:25.186638 27982 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:42:25.188438 27982 webserver.cc:460] Webserver started at http://127.30.33.2:37563/ using document root <none> and password file <none>
I20260810 23:42:25.189194 27982 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:25.189329 27982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:25.189661 27982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:42:25.192910 27982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "d6115409b66e42ce8731c180d1ece2cc"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "22572741899e2a110ed0b6b010533375"
server_key_iv: "e1e7d2a9e3cb689329bfa18997a13f70"
server_key_version: "encryptionkey@0"
I20260810 23:42:25.193732 27982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "d6115409b66e42ce8731c180d1ece2cc"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "22572741899e2a110ed0b6b010533375"
server_key_iv: "e1e7d2a9e3cb689329bfa18997a13f70"
server_key_version: "encryptionkey@0"
I20260810 23:42:25.199616 27982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:42:25.203430 28021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:25.205114 27982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:42:25.205322 27982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "d6115409b66e42ce8731c180d1ece2cc"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "22572741899e2a110ed0b6b010533375"
server_key_iv: "e1e7d2a9e3cb689329bfa18997a13f70"
server_key_version: "encryptionkey@0"
I20260810 23:42:25.205559 27982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:25.226440 27982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:25.231251 27982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:25.231597 27982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:25.232695 27982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:25.234127 27982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:25.234217 27982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:25.234354 27982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:25.234418 27982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:25.267735 27982 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:33001
I20260810 23:42:25.267788 28263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:33001 every 8 connection(s)
I20260810 23:42:25.269176 27982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:42:25.273890 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 27982
I20260810 23:42:25.274101 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:42:25.274574 30852 external_mini_cluster.cc:1428] Setting key 087d0d6ba3b4003b24fa9c9a3a79195f
Aug 10 23:42:25 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405345, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:25.291780 28269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.272089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:45797 (local address 127.30.33.62:36009)
0810 23:42:25.272618 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:25.272626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:25.273763 (+ 1137us) server_negotiation.cc:407] Connection header received
0810 23:42:25.275827 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:25.275831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:25.275900 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:25.275993 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:25.277874 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.278617 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.280358 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.280812 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.280929 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.285049 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:25.285055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:25.285058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:25.285102 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:25.288087 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:25.289057 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:25.289062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:25.289064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:25.289236 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:25.289752 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:25.289757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:25.289759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:25.290068 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:25.290221 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:25.291061 (+ 840us) server_negotiation.cc:299] Negotiation successful
0810 23:42:25.291325 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
I20260810 23:42:25.292841 28271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.272715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:45797)
0810 23:42:25.273559 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:25.273607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:25.275522 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:25.276319 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:25.276332 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:25.277037 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:25.277677 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.277696 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.278794 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.278799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.280187 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.280202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.281082 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.281100 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.281327 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.282111 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:25.282144 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:25.284739 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:25.288317 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:25.288327 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:25.288343 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:25.288883 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:25.289373 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:25.289377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:25.289380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:25.289606 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:25.290229 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:25.290239 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:25.290650 (+ 411us) client_negotiation.cc:769] Sending connection context
0810 23:42:25.290927 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 23:42:25.291585 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":346,"threads_started":1}
I20260810 23:42:25.294467 28264 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36009
I20260810 23:42:25.294857 28264 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:25.295984 28264 heartbeater.cc:511] Master 127.30.33.62:36009 requested a full tablet report, sending...
I20260810 23:42:25.297765 27596 ts_manager.cc:194] Registered new tserver with Master: d6115409b66e42ce8731c180d1ece2cc (127.30.33.2:33001)
I20260810 23:42:25.298611 27596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:45797
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:42:25.347355 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36009
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:25.703053 28274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:25.703366 28274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:25.703533 28274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:25.711511 28274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:25.711649 28274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:25.711812 28274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:42:25.722087 28274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:25.723536 28274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:25.725000 28274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:42:25.732241 27973 heartbeater.cc:503] Master 127.30.33.62:36009 was elected leader, sending a full tablet report...
W20260810 23:42:25.739651 28281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:25.739732 28284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:25.739699 28282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:25.741055 28274 server_base.cc:1034] running on GCE node
I20260810 23:42:25.741739 28274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:25.742745 28274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:25.743950 28274 hybrid_clock.cc:648] HybridClock initialized: now 1786405345743895 us; error 50 us; skew 500 ppm
I20260810 23:42:25.744282 28274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:25 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405345, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:25.747687 28274 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:42:25.749264 28274 webserver.cc:460] Webserver started at http://127.30.33.3:37091/ using document root <none> and password file <none>
I20260810 23:42:25.749979 28274 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:25.750087 28274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:25.750408 28274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:42:25.753917 28274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "9189b1c8a5ac4ab4814339d97153803f"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "46b52db37283b48f11c465dcf9c2f57f"
server_key_iv: "62f2d444687521326436f9574c8091dd"
server_key_version: "encryptionkey@0"
I20260810 23:42:25.754724 28274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "9189b1c8a5ac4ab4814339d97153803f"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "46b52db37283b48f11c465dcf9c2f57f"
server_key_iv: "62f2d444687521326436f9574c8091dd"
server_key_version: "encryptionkey@0"
I20260810 23:42:25.760591 28274 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:42:25.764515 28308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:25.766140 28274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:42:25.766347 28274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "9189b1c8a5ac4ab4814339d97153803f"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "46b52db37283b48f11c465dcf9c2f57f"
server_key_iv: "62f2d444687521326436f9574c8091dd"
server_key_version: "encryptionkey@0"
I20260810 23:42:25.766556 28274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:25.780942 28274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:25.785789 28274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:25.786116 28274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:25.787186 28274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:25.788689 28274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:25.788772 28274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:25.788900 28274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:25.788959 28274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:25.821947 28274 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:43507
I20260810 23:42:25.821983 28501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:43507 every 8 connection(s)
I20260810 23:42:25.823226 28274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:42:25.830806 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 28274
I20260810 23:42:25.831002 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:42:25.831611 30852 external_mini_cluster.cc:1428] Setting key 6c9f079958a99ea53bee4ff6d3e8df55
Aug 10 23:42:25 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405345, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:25.845016 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.825890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:54317 (local address 127.30.33.62:36009)
0810 23:42:25.826390 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:25.826397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:25.827479 (+ 1082us) server_negotiation.cc:407] Connection header received
0810 23:42:25.829434 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:25.829440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:25.829530 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:25.829631 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:25.831505 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.832400 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.834540 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.835077 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.835223 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.839032 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:25.839038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:25.839042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:25.839081 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:25.841885 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:25.842702 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:25.842705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:25.842707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:25.842830 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:25.843229 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:25.843232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:25.843234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:25.843568 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:25.843691 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:25.844532 (+ 841us) server_negotiation.cc:299] Negotiation successful
0810 23:42:25.844723 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":221,"threads_started":1}
I20260810 23:42:25.846424 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.826499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:54317)
0810 23:42:25.827268 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:25.827309 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:25.829112 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:25.829911 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:25.829920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:25.830533 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:25.831228 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.831259 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.832572 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.832578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.834355 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.834372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.835318 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.835324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.835601 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.836329 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:25.836357 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:25.838773 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:25.842067 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:25.842077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:25.842102 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:25.842591 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:25.842920 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:25.842923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:25.842925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:25.843134 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:25.843739 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:25.843747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:25.844151 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 23:42:25.844444 (+ 293us) client_negotiation.cc:240] Negotiation successful
0810 23:42:25.844914 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":283,"threads_started":1}
I20260810 23:42:25.847980 28505 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36009
I20260810 23:42:25.848369 28505 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:25.849211 28505 heartbeater.cc:511] Master 127.30.33.62:36009 requested a full tablet report, sending...
I20260810 23:42:25.850888 27595 ts_manager.cc:194] Registered new tserver with Master: 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507)
I20260810 23:42:25.851935 27595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:54317
I20260810 23:42:25.860993 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:42:25.880112 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.865899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.30.33.62:36009)
0810 23:42:25.866086 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:25.866090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:25.866520 (+ 430us) server_negotiation.cc:407] Connection header received
0810 23:42:25.866683 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:25.866686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:25.866762 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:25.866856 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:25.868585 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.869357 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.870978 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.871403 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.871592 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.874199 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:25.874204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:25.874207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:25.874257 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:25.877168 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:25.878036 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:25.878042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:25.878046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:25.878207 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:25.878645 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:25.878648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:25.878650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:25.879006 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:25.879206 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:25.879651 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 23:42:25.879866 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:42:25.884948 27595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:42:25.892630 27595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:42:25.933584 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.907277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:46632)
0810 23:42:25.918896 (+ 11619us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:25.918946 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:25.919203 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:25.924987 (+ 5784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:25.924997 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:25.925036 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:25.925507 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.925547 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.927626 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.927632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.929938 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.929953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.932523 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.932530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.932714 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.932781 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 23:42:25.932909 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:42:25.933139 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11308,"mutex_wait_us":102,"thread_start_us":15251,"threads_started":1}
I20260810 23:42:25.933934 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.920410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.30.33.3:43507)
0810 23:42:25.925645 (+ 5235us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:25.925655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:25.925694 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:42:25.925801 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:25.925809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:25.926057 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:25.926273 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:25.926958 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.928469 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.930728 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.932153 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.933075 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.933345 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:25.933424 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:42:25.933606 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5015,"thread_start_us":4833,"threads_started":1}
I20260810 23:42:25.934571 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.920024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43507 (local address 127.0.0.1:46152)
0810 23:42:25.923297 (+ 3273us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:25.923316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:25.923448 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:25.926294 (+ 2846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:25.926300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:25.926330 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:25.926705 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.926718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.928652 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.928657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.930484 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.930500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.932409 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.932416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.932667 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.932770 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 23:42:25.932893 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:42:25.934179 (+ 1286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3099,"mutex_wait_us":1079,"thread_start_us":3257,"threads_started":1}
I20260810 23:42:25.935154 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.907583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.30.33.1:34165)
0810 23:42:25.924140 (+ 16557us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:25.924150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:25.924185 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:42:25.924326 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:25.924334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:25.924621 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:25.924844 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:25.925774 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.927391 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.930196 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.931941 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.932089 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.932378 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:25.934617 (+ 2239us) server_negotiation.cc:299] Negotiation successful
0810 23:42:25.934822 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16352,"thread_start_us":16638,"threads_started":1}
I20260810 23:42:25.935262 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.907410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:33001 (local address 127.0.0.1:57750)
0810 23:42:25.922907 (+ 15497us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:25.922938 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:25.923154 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:25.924558 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:25.924566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:25.924612 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:25.925053 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.925069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.927036 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.927041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.929935 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:25.929953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.934422 (+ 4469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.934429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:25.934613 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.934648 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:42:25.934743 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:42:25.934883 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15314,"mutex_wait_us":35,"thread_start_us":10991,"threads_started":1}
I20260810 23:42:25.935338 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:25.907773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.30.33.2:33001)
0810 23:42:25.920056 (+ 12283us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:25.920068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:25.923088 (+ 3020us) server_negotiation.cc:407] Connection header received
0810 23:42:25.923379 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:25.923390 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:25.923762 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:25.924008 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:25.925298 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.926898 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.931724 (+ 4826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:25.934177 (+ 2453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:25.934327 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:25.934489 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:25.934842 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 23:42:25.935025 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12059,"thread_start_us":11879,"threads_started":1}
I20260810 23:42:25.939049 28401 tablet_service.cc:1467] Processing CreateTablet for tablet a2bab3e65b364b1187cb059a10946ef7 (DEFAULT_TABLE table=test-table [id=a66416b1d8444f74af3168a80d9f3dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:25.939065 27844 tablet_service.cc:1467] Processing CreateTablet for tablet a2bab3e65b364b1187cb059a10946ef7 (DEFAULT_TABLE table=test-table [id=a66416b1d8444f74af3168a80d9f3dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:25.939065 28117 tablet_service.cc:1467] Processing CreateTablet for tablet a2bab3e65b364b1187cb059a10946ef7 (DEFAULT_TABLE table=test-table [id=a66416b1d8444f74af3168a80d9f3dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:25.940606 28401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bab3e65b364b1187cb059a10946ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:25.940625 27844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bab3e65b364b1187cb059a10946ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:25.940654 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bab3e65b364b1187cb059a10946ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:25.953809 28545 tablet_bootstrap.cc:492] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Bootstrap starting.
I20260810 23:42:25.956475 28546 tablet_bootstrap.cc:492] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Bootstrap starting.
I20260810 23:42:25.958842 28545 tablet_bootstrap.cc:654] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:25.959594 28547 tablet_bootstrap.cc:492] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Bootstrap starting.
I20260810 23:42:25.960602 28546 tablet_bootstrap.cc:654] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:25.960646 28545 log.cc:824] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:25.962144 28546 log.cc:824] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:25.963061 28547 tablet_bootstrap.cc:654] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:25.964510 28547 log.cc:824] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:25.966715 28545 tablet_bootstrap.cc:492] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: No bootstrap required, opened a new log
I20260810 23:42:25.966715 28546 tablet_bootstrap.cc:492] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: No bootstrap required, opened a new log
I20260810 23:42:25.967059 28546 ts_tablet_manager.cc:1397] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260810 23:42:25.967060 28545 ts_tablet_manager.cc:1397] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260810 23:42:25.967723 28547 tablet_bootstrap.cc:492] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: No bootstrap required, opened a new log
I20260810 23:42:25.968072 28547 ts_tablet_manager.cc:1397] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260810 23:42:25.976199 28547 raft_consensus.cc:348] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:25.976653 28547 raft_consensus.cc:374] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:25.976780 28547 raft_consensus.cc:729] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5a2ce46b444e07acd9b9096c47e084, State: Initialized, Role: FOLLOWER
I20260810 23:42:25.977483 28547 consensus_queue.cc:260] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:25.978559 28545 raft_consensus.cc:348] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:25.978549 28546 raft_consensus.cc:348] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:25.979045 28546 raft_consensus.cc:374] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:25.979135 28545 raft_consensus.cc:374] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:25.979167 28546 raft_consensus.cc:729] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6115409b66e42ce8731c180d1ece2cc, State: Initialized, Role: FOLLOWER
I20260810 23:42:25.979292 28545 raft_consensus.cc:729] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9189b1c8a5ac4ab4814339d97153803f, State: Initialized, Role: FOLLOWER
I20260810 23:42:25.979916 28546 consensus_queue.cc:260] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:25.980245 28545 consensus_queue.cc:260] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:25.981283 28264 heartbeater.cc:503] Master 127.30.33.62:36009 was elected leader, sending a full tablet report...
I20260810 23:42:25.982010 28546 ts_tablet_manager.cc:1428] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260810 23:42:25.983392 28505 heartbeater.cc:503] Master 127.30.33.62:36009 was elected leader, sending a full tablet report...
I20260810 23:42:25.984154 28545 ts_tablet_manager.cc:1428] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260810 23:42:25.986485 28547 ts_tablet_manager.cc:1428] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
W20260810 23:42:26.021430 28265 tablet.cc:2369] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:42:26.075011 28506 tablet.cc:2369] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:42:26.188876 28557 raft_consensus.cc:482] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:42:26.189208 28557 raft_consensus.cc:504] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:26.191203 28557 leader_election.cc:288] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165), 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507)
I20260810 23:42:26.198546 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.191760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:46617 (local address 127.30.33.3:43507)
0810 23:42:26.191948 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:26.191954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:26.192642 (+ 688us) server_negotiation.cc:407] Connection header received
0810 23:42:26.192795 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:26.192799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:26.192863 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:26.192956 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:26.193634 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.194386 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.196570 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.197591 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.197787 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.197895 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:26.198156 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:42:26.198292 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:42:26.198556 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.191659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:38871 (local address 127.30.33.1:34165)
0810 23:42:26.191876 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:26.191883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:26.192498 (+ 615us) server_negotiation.cc:407] Connection header received
0810 23:42:26.192725 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:26.192731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:26.192813 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:26.192882 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:26.193606 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.194382 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.196586 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.197413 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.197539 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.197633 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:26.198156 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 23:42:26.198306 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:42:26.198742 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.191433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.30.33.2:38871)
0810 23:42:26.192347 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:26.192371 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:26.192561 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:26.192936 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:26.192940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:26.192965 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:26.193444 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:26.193457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.194540 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.194544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:26.196379 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:26.196394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.197720 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.197728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:26.197945 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.197975 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:42:26.198064 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:42:26.198228 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":90,"thread_start_us":470,"threads_started":1}
I20260810 23:42:26.198805 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.191728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43507 (local address 127.30.33.2:46617)
0810 23:42:26.192511 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:26.192528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:26.192668 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:26.192990 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:26.192996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:26.193020 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:26.193466 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:26.193477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.194537 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.194541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:26.196365 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:26.196383 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.197848 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.197853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:26.197987 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.198012 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:42:26.198115 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:42:26.198255 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":36,"thread_start_us":467,"threads_started":1}
I20260810 23:42:26.200006 27894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bab3e65b364b1187cb059a10946ef7" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5a2ce46b444e07acd9b9096c47e084" is_pre_election: true
I20260810 23:42:26.200029 28431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bab3e65b364b1187cb059a10946ef7" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9189b1c8a5ac4ab4814339d97153803f" is_pre_election: true
I20260810 23:42:26.200554 27894 raft_consensus.cc:2454] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 0.
I20260810 23:42:26.200547 28431 raft_consensus.cc:2454] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 0.
I20260810 23:42:26.201298 28025 leader_election.cc:302] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5a2ce46b444e07acd9b9096c47e084, d6115409b66e42ce8731c180d1ece2cc; no voters:
I20260810 23:42:26.201728 28557 raft_consensus.cc:2781] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 23:42:26.201821 27975 tablet.cc:2369] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:42:26.201896 28557 raft_consensus.cc:482] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:42:26.201992 28557 raft_consensus.cc:3037] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:26.204422 28557 raft_consensus.cc:504] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:26.205129 28557 leader_election.cc:288] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 election: Requested vote from peers 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165), 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507)
I20260810 23:42:26.205732 27894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bab3e65b364b1187cb059a10946ef7" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5a2ce46b444e07acd9b9096c47e084"
I20260810 23:42:26.205755 28431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bab3e65b364b1187cb059a10946ef7" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9189b1c8a5ac4ab4814339d97153803f"
I20260810 23:42:26.205952 27894 raft_consensus.cc:3037] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:26.205952 28431 raft_consensus.cc:3037] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:26.208314 28431 raft_consensus.cc:2454] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 1.
I20260810 23:42:26.208317 27894 raft_consensus.cc:2454] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 1.
I20260810 23:42:26.208870 28023 leader_election.cc:302] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a5a2ce46b444e07acd9b9096c47e084, 9189b1c8a5ac4ab4814339d97153803f, d6115409b66e42ce8731c180d1ece2cc; no voters:
I20260810 23:42:26.209266 28557 raft_consensus.cc:2781] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:42:26.209749 28557 raft_consensus.cc:686] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 1 LEADER]: Becoming Leader. State: Replica: d6115409b66e42ce8731c180d1ece2cc, State: Running, Role: LEADER
I20260810 23:42:26.210333 28557 consensus_queue.cc:237] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } }
I20260810 23:42:26.215718 27594 catalog_manager.cc:5495] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc reported cstate change: term changed from 0 to 1, leader changed from <none> to d6115409b66e42ce8731c180d1ece2cc (127.30.33.2). New cstate: current_term: 1 leader_uuid: "d6115409b66e42ce8731c180d1ece2cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } health_report { overall_health: UNKNOWN } } }
I20260810 23:42:26.238454 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.232532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.30.33.2:33001)
0810 23:42:26.232766 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:26.232770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:26.232794 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:42:26.233009 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:26.233013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:26.233127 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:26.233266 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:42:26.233701 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.234469 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.236136 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.236569 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.236679 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.237189 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:42:26.237752 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:42:26.237865 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:26.237985 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:42:26.238064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:42:26.247417 27894 raft_consensus.cc:1261] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Refusing update from remote peer d6115409b66e42ce8731c180d1ece2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:26.247416 28431 raft_consensus.cc:1261] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Refusing update from remote peer d6115409b66e42ce8731c180d1ece2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:26.248590 28562 consensus_queue.cc:1035] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:26.248961 28557 consensus_queue.cc:1035] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:26.261548 28567 mvcc.cc:205] Tried to move back new op lower bound from 7317116298218246144 to 7317116298081529856. Current Snapshot: MvccSnapshot[applied={T|T < 7317116298218246144}]
I20260810 23:42:26.284845 27594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:26.285125 27594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:26.288815 27594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Sending DeleteTablet for 3 replicas of tablet a2bab3e65b364b1187cb059a10946ef7
I20260810 23:42:26.290043 27844 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bab3e65b364b1187cb059a10946ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:26 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:46632
I20260810 23:42:26.290055 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bab3e65b364b1187cb059a10946ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:26 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:57750
I20260810 23:42:26.295128 28401 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bab3e65b364b1187cb059a10946ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:26 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:46152
I20260810 23:42:26.304059 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 27716
I20260810 23:42:26.313114 28594 tablet_replica.cc:330] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: stopping tablet replica
I20260810 23:42:26.313934 28594 raft_consensus.cc:2229] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:26.314736 28594 raft_consensus.cc:2258] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:26.316157 28593 tablet_replica.cc:330] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: stopping tablet replica
I20260810 23:42:26.316653 28593 raft_consensus.cc:2229] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:42:26.318028 28594 ts_tablet_manager.cc:1905] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:26.319679 28593 raft_consensus.cc:2258] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:26.320335 28592 tablet_replica.cc:330] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: stopping tablet replica
I20260810 23:42:26.320979 28592 raft_consensus.cc:2229] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:26.321640 28592 raft_consensus.cc:2258] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:26.324652 28593 ts_tablet_manager.cc:1905] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:26.325045 28594 ts_tablet_manager.cc:1918] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:26.325290 28594 log.cc:1196] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/a2bab3e65b364b1187cb059a10946ef7
I20260810 23:42:26.326050 28594 ts_tablet_manager.cc:1939] T a2bab3e65b364b1187cb059a10946ef7 P 9189b1c8a5ac4ab4814339d97153803f: Deleting consensus metadata
I20260810 23:42:26.327370 28592 ts_tablet_manager.cc:1905] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:26.328347 27567 catalog_manager.cc:4841] TS 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507): tablet a2bab3e65b364b1187cb059a10946ef7 (table test-table [id=a66416b1d8444f74af3168a80d9f3dd1]) successfully deleted
Aug 10 23:42:26 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:42:26.335319 28592 ts_tablet_manager.cc:1918] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:26.335553 28592 log.cc:1196] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/a2bab3e65b364b1187cb059a10946ef7
I20260810 23:42:26.337157 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.305532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.30.33.1:34165)
0810 23:42:26.305751 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:26.305757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:26.316462 (+ 10705us) server_negotiation.cc:407] Connection header received
0810 23:42:26.316702 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:26.316708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:26.316812 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:26.316924 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:26.319007 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.320199 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.322255 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.322896 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.323008 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.329878 (+ 6870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:26.329892 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:26.329910 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:26.329958 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:26.333300 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:26.334244 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:26.334255 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:26.334259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:26.334459 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:26.334946 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:26.334950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:26.334952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:26.335378 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:26.335773 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:26.336359 (+ 586us) server_negotiation.cc:299] Negotiation successful
0810 23:42:26.336621 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:42:26.340286 28592 ts_tablet_manager.cc:1939] T a2bab3e65b364b1187cb059a10946ef7 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting consensus metadata
I20260810 23:42:26.340768 27966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:26.342918 28593 ts_tablet_manager.cc:1918] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:26.343142 28593 log.cc:1196] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/a2bab3e65b364b1187cb059a10946ef7
I20260810 23:42:26.343832 28593 ts_tablet_manager.cc:1939] T a2bab3e65b364b1187cb059a10946ef7 P d6115409b66e42ce8731c180d1ece2cc: Deleting consensus metadata
I20260810 23:42:26.345973 27567 catalog_manager.cc:4841] TS d6115409b66e42ce8731c180d1ece2cc (127.30.33.2:33001): tablet a2bab3e65b364b1187cb059a10946ef7 (table test-table [id=a66416b1d8444f74af3168a80d9f3dd1]) successfully deleted
I20260810 23:42:26.435614 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 27716
I20260810 23:42:26.435745 27569 catalog_manager.cc:4841] TS 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165): tablet a2bab3e65b364b1187cb059a10946ef7 (table test-table [id=a66416b1d8444f74af3168a80d9f3dd1]) successfully deleted
I20260810 23:42:26.466701 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 27982
Aug 10 23:42:26 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:42:26.485180 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.467667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.30.33.2:33001)
0810 23:42:26.467899 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:26.467904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:26.467925 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:42:26.468068 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:26.468074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:26.468149 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:26.468240 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:26.469632 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.470600 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.472049 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.472508 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.472638 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.477373 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:26.477390 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:26.477411 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:26.477467 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:26.481783 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:26.482759 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:26.482768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:26.482770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:26.482928 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:26.483400 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:26.483404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:26.483406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:26.483870 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:26.484235 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:26.484569 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:42:26.484811 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 23:42:26.485961 28258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:26.603390 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 27982
I20260810 23:42:26.629928 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 28274
Aug 10 23:42:26 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:42:26.646243 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:26.630788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.30.33.3:43507)
0810 23:42:26.630983 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:26.630989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:26.631009 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:42:26.631190 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:26.631195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:26.631275 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:26.631361 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:26.632750 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.633571 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.634907 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:26.635295 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:26.635408 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:26.640099 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:26.640116 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:26.640134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:26.640177 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:26.642800 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:26.643678 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:26.643688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:26.643690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:26.643854 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:26.644291 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:26.644295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:26.644297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:26.644828 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:26.645181 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:26.645623 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 23:42:26.645869 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:42:26.647048 28494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:26.736764 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 28274
I20260810 23:42:26.758870 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 27540
I20260810 23:42:26.760345 27657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:26.876094 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 27540
I20260810 23:42:46.907703 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36009
--webserver_interface=127.30.33.62
--webserver_port=41751
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:42:47.277333 28608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:47.277673 28608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:47.277794 28608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:47.285293 28608 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:42:47.285422 28608 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:42:47.285462 28608 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:42:47.285534 28608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:42:47.285650 28608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:47.285735 28608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:47.285799 28608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:42:47.285867 28608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:42:47.296720 28608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:47.298118 28608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:47.299537 28608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:47.326972 28622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:47.327098 28628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:47.326995 28623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:47.327378 28608 server_base.cc:1034] running on GCE node
I20260810 23:42:47.328699 28608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:47.330539 28608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:47.331810 28608 hybrid_clock.cc:648] HybridClock initialized: now 1786405367331710 us; error 63 us; skew 500 ppm
I20260810 23:42:47.332340 28608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:47 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405367, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:47.340021 28608 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:42:47.342065 28608 webserver.cc:460] Webserver started at http://127.30.33.62:41751/ using document root <none> and password file <none>
I20260810 23:42:47.342999 28608 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:47.343154 28608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:47.350391 28608 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:42:47.354409 28643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:47.356335 28608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 23:42:47.356572 28608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "9d31291a6aec436db87c30f3fb15f59b"
format_stamp: "Formatted at 2026-08-10 23:42:23 on dist-test-slave-pww0"
server_key: "bc15556cd5023f023deef70e0f3da139"
server_key_iv: "910405615564914e68c9ea54838158be"
server_key_version: "encryptionkey@0"
I20260810 23:42:47.357764 28608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:47.395659 28608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:47.400501 28608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:47.400827 28608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:47.424541 28608 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:36009
I20260810 23:42:47.424541 28723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:36009 every 8 connection(s)
I20260810 23:42:47.426741 28608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:42:47.432197 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 28608
I20260810 23:42:47.433260 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:34165
--local_ip_for_outbound_sockets=127.30.33.1
--tserver_master_addrs=127.30.33.62:36009
--webserver_port=46805
--webserver_interface=127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:42:47.436139 28724 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:42:47.441107 28724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Bootstrap starting.
I20260810 23:42:47.465190 28724 log.cc:824] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:47.483060 28724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:42:47.483878 28724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Bootstrap complete.
I20260810 23:42:47.499366 28724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:47.500829 28724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d31291a6aec436db87c30f3fb15f59b, State: Initialized, Role: FOLLOWER
I20260810 23:42:47.501701 28724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:47.502063 28724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:42:47.502207 28724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:42:47.502422 28724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:42:47.508833 28724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:47.509690 28724 leader_election.cc:302] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d31291a6aec436db87c30f3fb15f59b; no voters:
I20260810 23:42:47.510500 28724 leader_election.cc:288] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:42:47.510672 28743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:42:47.520673 28724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:42:47.523427 28743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 LEADER]: Becoming Leader. State: Replica: 9d31291a6aec436db87c30f3fb15f59b, State: Running, Role: LEADER
I20260810 23:42:47.524300 28743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:42:47.527606 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d31291a6aec436db87c30f3fb15f59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } } }
I20260810 23:42:47.528116 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:47.527630 28748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d31291a6aec436db87c30f3fb15f59b. Latest consensus state: current_term: 2 leader_uuid: "9d31291a6aec436db87c30f3fb15f59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } } }
I20260810 23:42:47.528610 28748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: This master's current role is: LEADER
I20260810 23:42:47.536653 28756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:42:47.558876 28756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:42:47.560761 28756 catalog_manager.cc:1259] Loaded cluster ID: 7add57300a114fc886eecd1763a8ebc3
I20260810 23:42:47.560992 28756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:42:47.566453 28756 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:42:47.568697 28756 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Loaded TSK: 0
I20260810 23:42:47.588882 28779 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:42:47.601435 28756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Generated new TSK 1
I20260810 23:42:47.601948 28756 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:42:47.866784 28734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:47.867147 28734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:47.867302 28734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:47.876302 28734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:47.876461 28734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:47.876657 28734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:42:47.887230 28734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:34165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=46805
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:47.888715 28734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:47.890242 28734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:47.905732 28791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:47.906472 28790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:47.906497 28795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:47.907064 28734 server_base.cc:1034] running on GCE node
I20260810 23:42:47.907831 28734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:47.908957 28734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:47.910223 28734 hybrid_clock.cc:648] HybridClock initialized: now 1786405367910167 us; error 49 us; skew 500 ppm
I20260810 23:42:47.910622 28734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:47 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405367, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:47.913954 28734 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:42:47.915800 28734 webserver.cc:460] Webserver started at http://127.30.33.1:46805/ using document root <none> and password file <none>
I20260810 23:42:47.916555 28734 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:47.916697 28734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:47.922904 28734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:42:47.926204 28809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:47.927871 28734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:42:47.928061 28734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "3a5a2ce46b444e07acd9b9096c47e084"
format_stamp: "Formatted at 2026-08-10 23:42:24 on dist-test-slave-pww0"
server_key: "bdeabde505ed412e2efb187dbdb51143"
server_key_iv: "ab5a1352dd98aa362151170326ccd128"
server_key_version: "encryptionkey@0"
I20260810 23:42:47.929167 28734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:47.944047 28734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:47.948874 28734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:47.949131 28734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:47.950335 28734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:47.952023 28734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:47.952142 28734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:47.952255 28734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:47.952327 28734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:47.987185 28734 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:34165
I20260810 23:42:47.987277 29006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:34165 every 8 connection(s)
I20260810 23:42:47.989008 28734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:42:47.996491 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 28734
I20260810 23:42:47.997311 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:33001
--local_ip_for_outbound_sockets=127.30.33.2
--tserver_master_addrs=127.30.33.62:36009
--webserver_port=37563
--webserver_interface=127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:42:48 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405367, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:48.017549 29013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:47.992380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:43115)
0810 23:42:47.993262 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:47.993313 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:47.995245 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:47.997075 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:47.997087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:47.997893 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:47.998733 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:47.998763 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.000735 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.000741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:48.002572 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:48.002594 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.003935 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.003942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:48.004241 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:48.005093 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:48.005123 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:48.008181 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:48.013083 (+ 4902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:48.013092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:48.013107 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:48.013623 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:48.014203 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:48.014206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:48.014208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:48.014413 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:48.015420 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:48.015426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:48.015810 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 23:42:48.016058 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 23:42:48.016500 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":358,"threads_started":1}
I20260810 23:42:48.017943 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:47.992230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:43115 (local address 127.30.33.62:36009)
0810 23:42:47.992848 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:47.992861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:47.993495 (+ 634us) server_negotiation.cc:407] Connection header received
0810 23:42:47.995954 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:47.995970 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:47.996378 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:47.996779 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:47.999048 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.000508 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.002801 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.003644 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.003803 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:48.008589 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:48.008599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:48.008638 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:48.008699 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:48.012851 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.013823 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.013834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.013842 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.014089 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.014522 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.014526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.014530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.015287 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:48.015514 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:48.016352 (+ 838us) server_negotiation.cc:299] Negotiation successful
0810 23:42:48.016807 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":265,"threads_started":1}
I20260810 23:42:48.021312 29007 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36009
I20260810 23:42:48.021684 29007 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:48.022558 29007 heartbeater.cc:511] Master 127.30.33.62:36009 requested a full tablet report, sending...
I20260810 23:42:48.025110 28667 ts_manager.cc:194] Registered new tserver with Master: 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165)
I20260810 23:42:48.026744 28667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:43115
W20260810 23:42:48.359673 29014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:48.360015 29014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:48.360172 29014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:48.367916 29014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:48.368072 29014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:48.368257 29014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:42:48.378731 29014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:33001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=37563
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:48.380250 29014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:48.381735 29014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:48.396006 29024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:48.396142 29022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:48.396035 29031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:48.396793 29014 server_base.cc:1034] running on GCE node
I20260810 23:42:48.397537 29014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:48.398581 29014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:48.399791 29014 hybrid_clock.cc:648] HybridClock initialized: now 1786405368399734 us; error 54 us; skew 500 ppm
I20260810 23:42:48.400134 29014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:48 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405368, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:48.403491 29014 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:42:48.405102 29014 webserver.cc:460] Webserver started at http://127.30.33.2:37563/ using document root <none> and password file <none>
I20260810 23:42:48.405870 29014 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:48.406024 29014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:48.412039 29014 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:42:48.414969 29050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:48.416500 29014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:42:48.416687 29014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "d6115409b66e42ce8731c180d1ece2cc"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "22572741899e2a110ed0b6b010533375"
server_key_iv: "e1e7d2a9e3cb689329bfa18997a13f70"
server_key_version: "encryptionkey@0"
I20260810 23:42:48.417747 29014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:48.436539 29014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:48.441071 29014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:48.441349 29014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:48.442385 29014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:48.443837 29014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:48.443948 29014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:48.444056 29014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:48.444130 29014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:48.476707 29014 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:33001
I20260810 23:42:48.476768 29261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:33001 every 8 connection(s)
I20260810 23:42:48.478255 29014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:42:48.485236 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 29014
I20260810 23:42:48.486058 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:43507
--local_ip_for_outbound_sockets=127.30.33.3
--tserver_master_addrs=127.30.33.62:36009
--webserver_port=37091
--webserver_interface=127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:42:48 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405368, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:48.502091 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:48.480964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:43523 (local address 127.30.33.62:36009)
0810 23:42:48.481204 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:48.481211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:48.482511 (+ 1300us) server_negotiation.cc:407] Connection header received
0810 23:42:48.484494 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:48.484499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:48.484598 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:48.484697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:48.487099 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.487928 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.490223 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.490682 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.490835 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:48.495589 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:48.495597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:48.495600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:48.495645 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:48.498384 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.499495 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.499501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.499505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.499697 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.500170 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.500174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.500178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.500579 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:48.500737 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:48.501472 (+ 735us) server_negotiation.cc:299] Negotiation successful
0810 23:42:48.501738 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:42:48.503070 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:48.481492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:43523)
0810 23:42:48.482308 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:48.482371 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:48.484191 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:48.485079 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:48.485093 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:48.485944 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:48.486874 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:48.486903 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.488113 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.488120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:48.490003 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:48.490023 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.491085 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.491094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:48.491412 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:48.492246 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:48.492278 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:48.495208 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:48.498591 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:48.498602 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:48.498630 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:48.499296 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:48.499827 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:48.499832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:48.499834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:48.500060 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:48.500708 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:48.500715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:48.501081 (+ 366us) client_negotiation.cc:769] Sending connection context
0810 23:42:48.501333 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 23:42:48.501792 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":336,"threads_started":1}
I20260810 23:42:48.504664 29263 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36009
I20260810 23:42:48.504994 29263 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:48.505816 29263 heartbeater.cc:511] Master 127.30.33.62:36009 requested a full tablet report, sending...
I20260810 23:42:48.507525 28667 ts_manager.cc:194] Registered new tserver with Master: d6115409b66e42ce8731c180d1ece2cc (127.30.33.2:33001)
I20260810 23:42:48.508391 28667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:43523
W20260810 23:42:48.840804 29267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:42:48.841138 29267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:42:48.841281 29267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:42:48.848780 29267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:42:48.848933 29267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:42:48.849086 29267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:42:48.859584 29267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:43507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=37091
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:42:48.861043 29267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:42:48.862550 29267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:42:48.876911 29297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:48.876967 29288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:42:48.877364 29291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:42:48.878122 29267 server_base.cc:1034] running on GCE node
I20260810 23:42:48.878867 29267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:42:48.879983 29267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:42:48.881220 29267 hybrid_clock.cc:648] HybridClock initialized: now 1786405368881149 us; error 58 us; skew 500 ppm
I20260810 23:42:48.881608 29267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:42:48 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405368, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:42:48.885149 29267 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:42:48.886693 29267 webserver.cc:460] Webserver started at http://127.30.33.3:37091/ using document root <none> and password file <none>
I20260810 23:42:48.887415 29267 fs_manager.cc:359] Metadata directory not provided
I20260810 23:42:48.887594 29267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:42:48.893491 29267 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:42:48.896708 29310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:48.898288 29267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:42:48.898479 29267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "9189b1c8a5ac4ab4814339d97153803f"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "46b52db37283b48f11c465dcf9c2f57f"
server_key_iv: "62f2d444687521326436f9574c8091dd"
server_key_version: "encryptionkey@0"
I20260810 23:42:48.899569 29267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:42:48.916213 29267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:42:48.920801 29267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:42:48.921083 29267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:42:48.922267 29267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:42:48.923835 29267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:42:48.923936 29267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:48.924042 29267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:42:48.924113 29267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:42:48.956168 29267 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:43507
I20260810 23:42:48.956219 29520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:43507 every 8 connection(s)
I20260810 23:42:48.957733 29267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:42:48.961678 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 29267
Aug 10 23:42:48 dist-test-slave-pww0 krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405368, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:42:48.978921 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:48.962729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.30.33.62:36009)
0810 23:42:48.963218 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:48.963223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:48.963680 (+ 457us) server_negotiation.cc:407] Connection header received
0810 23:42:48.963841 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:48.963847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:48.963928 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:48.964079 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:48.965921 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.967040 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.969022 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.969678 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.969815 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:48.973231 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:48.973236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:48.973239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:48.973275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:48.976103 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.976825 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.976830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.976832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.976983 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.977391 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.977395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.977397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.977711 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:48.977979 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:48.978300 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 23:42:48.978527 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":233,"threads_started":1}
I20260810 23:42:48.981400 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:48.960547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:57011 (local address 127.30.33.62:36009)
0810 23:42:48.960788 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:48.960792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:48.962242 (+ 1450us) server_negotiation.cc:407] Connection header received
0810 23:42:48.964974 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:48.964981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:48.965065 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:48.965188 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:48.967290 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.968437 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.970542 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.970957 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.971050 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:48.975037 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:48.975042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:48.975046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:48.975091 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:48.977848 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.978622 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.978626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.978628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.978762 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:48.979338 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:48.979342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:48.979344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:48.979683 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:48.979817 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:48.980802 (+ 985us) server_negotiation.cc:299] Negotiation successful
0810 23:42:48.981038 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:42:48.982334 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:48.961030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:57011)
0810 23:42:48.962012 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:48.962082 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:48.964675 (+ 2593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:48.965470 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:48.965480 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:48.966213 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:42:48.967089 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:48.967118 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.968630 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.968634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:48.970409 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:48.970422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.971208 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:48.971216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:48.971571 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:48.972284 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:42:48.972318 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:42:48.974804 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:42:48.978022 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:48.978033 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:48.978048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:48.978493 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:48.978915 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:42:48.978920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:42:48.978924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:42:48.979208 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:42:48.979840 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:42:48.979848 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:42:48.980331 (+ 483us) client_negotiation.cc:769] Sending connection context
0810 23:42:48.980665 (+ 334us) client_negotiation.cc:240] Negotiation successful
0810 23:42:48.981098 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":373,"threads_started":1}
I20260810 23:42:48.983713 29521 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36009
I20260810 23:42:48.984025 29521 heartbeater.cc:464] Registering TS with master...
I20260810 23:42:48.984787 29521 heartbeater.cc:511] Master 127.30.33.62:36009 requested a full tablet report, sending...
I20260810 23:42:48.986261 28667 ts_manager.cc:194] Registered new tserver with Master: 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507)
I20260810 23:42:48.987164 28667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:57011
I20260810 23:42:48.994311 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:42:49.007382 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:48.995251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.30.33.62:36009)
0810 23:42:48.995429 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:48.995433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:48.995962 (+ 529us) server_negotiation.cc:407] Connection header received
0810 23:42:48.996163 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:48.996167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:48.996323 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:48.996453 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:42:48.996825 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:48.997585 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.003723 (+ 6138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.004557 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.004742 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.006192 (+ 1450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:42:49.006627 (+ 435us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 23:42:49.007087 (+ 460us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50790: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 23:42:49.007573 29010 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50790: token expired
W20260810 23:42:49.009155 28519 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.33.62:36009)
I20260810 23:42:49.023866 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.010075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.30.33.62:36009)
0810 23:42:49.010354 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.010359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.010383 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:42:49.010444 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.010449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.010523 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.010627 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:49.012627 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.013416 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.014883 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.015315 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.015408 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.018036 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:49.018041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:49.018045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:49.018080 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:49.021066 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.021875 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.021879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.021882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.022024 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.022441 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.022445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.022448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.022825 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:49.023061 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.023359 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.023611 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:42:49.030452 29007 heartbeater.cc:503] Master 127.30.33.62:36009 was elected leader, sending a full tablet report...
I20260810 23:42:49.027752 28667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:42:49.035838 28667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:42:49.061353 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.048774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:33001 (local address 127.0.0.1:34920)
0810 23:42:49.049641 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.049689 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:49.049901 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:49.051069 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:49.051079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:49.051150 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:49.051640 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.051656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.054141 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.054148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.057038 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.057074 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.060077 (+ 3003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.060086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.060414 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.060493 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 23:42:49.060670 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 23:42:49.060910 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":399,"threads_started":1}
I20260810 23:42:49.061985 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.049010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.30.33.2:33001)
0810 23:42:49.049661 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.049671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.049805 (+ 134us) server_negotiation.cc:407] Connection header received
0810 23:42:49.050225 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.050235 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.050534 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.050751 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:49.051995 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.053879 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.057405 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.059311 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.059537 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.059811 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.061420 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.061655 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":946,"threads_started":1}
I20260810 23:42:49.068039 29141 tablet_service.cc:1467] Processing CreateTablet for tablet 034f34fb8e724ec29739dade39b9d197 (DEFAULT_TABLE table=test-table [id=096604a6c8da446ba50a8f5f0a17df1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:49.069653 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034f34fb8e724ec29739dade39b9d197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:49.072075 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.050258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.30.33.1:34165)
0810 23:42:49.063501 (+ 13243us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.063509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.063539 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:42:49.063642 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.063650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.063972 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.064215 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:49.064870 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.066473 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.068969 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.070489 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.070652 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.070901 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.071556 (+ 655us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.071733 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13022,"thread_start_us":12812,"threads_started":1}
I20260810 23:42:49.072386 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.050695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.30.33.3:43507)
0810 23:42:49.063128 (+ 12433us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.063138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.063181 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:42:49.063305 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.063314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.063664 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.064477 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:49.064813 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.066089 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.068508 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.071032 (+ 2524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.071168 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.071394 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.071871 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.072044 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12219,"thread_start_us":12068,"threads_started":1}
I20260810 23:42:49.072892 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.050061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43507 (local address 127.0.0.1:54958)
0810 23:42:49.062312 (+ 12251us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.062335 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:49.062475 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:49.064005 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:49.064010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:49.064042 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:49.064263 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.064277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.066259 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.066263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.068289 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.068309 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.071207 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.071213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.071408 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.071442 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:42:49.071560 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:42:49.071723 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11878,"mutex_wait_us":57,"thread_start_us":13075,"threads_started":1}
I20260810 23:42:49.073350 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.049516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45516)
0810 23:42:49.060757 (+ 11241us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.060781 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:49.060915 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:49.064212 (+ 3297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:49.064217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:49.064245 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:49.064580 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.064632 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.066647 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.066652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.068765 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.068781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.070784 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.070792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.070995 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.071033 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:42:49.071142 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:42:49.071314 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11043,"mutex_wait_us":1,"thread_start_us":11463,"threads_started":1}
I20260810 23:42:49.077376 29405 tablet_service.cc:1467] Processing CreateTablet for tablet 034f34fb8e724ec29739dade39b9d197 (DEFAULT_TABLE table=test-table [id=096604a6c8da446ba50a8f5f0a17df1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:49.077438 28891 tablet_service.cc:1467] Processing CreateTablet for tablet 034f34fb8e724ec29739dade39b9d197 (DEFAULT_TABLE table=test-table [id=096604a6c8da446ba50a8f5f0a17df1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:42:49.078787 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034f34fb8e724ec29739dade39b9d197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:49.078943 28891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034f34fb8e724ec29739dade39b9d197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:42:49.086364 29550 tablet_bootstrap.cc:492] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Bootstrap starting.
I20260810 23:42:49.091138 29550 tablet_bootstrap.cc:654] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:49.092705 29550 log.cc:824] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:49.093793 29551 tablet_bootstrap.cc:492] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Bootstrap starting.
I20260810 23:42:49.094743 29552 tablet_bootstrap.cc:492] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Bootstrap starting.
I20260810 23:42:49.098330 29550 tablet_bootstrap.cc:492] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: No bootstrap required, opened a new log
I20260810 23:42:49.098677 29552 tablet_bootstrap.cc:654] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:49.098678 29550 ts_tablet_manager.cc:1397] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260810 23:42:49.099431 29551 tablet_bootstrap.cc:654] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Neither blocks nor log segments found. Creating new log.
I20260810 23:42:49.099817 29552 log.cc:824] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:49.100782 29551 log.cc:824] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Log is configured to *not* fsync() on all Append() calls
I20260810 23:42:49.103107 29552 tablet_bootstrap.cc:492] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: No bootstrap required, opened a new log
I20260810 23:42:49.103377 29552 ts_tablet_manager.cc:1397] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:42:49.104193 29551 tablet_bootstrap.cc:492] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: No bootstrap required, opened a new log
I20260810 23:42:49.104489 29551 ts_tablet_manager.cc:1397] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260810 23:42:49.109249 29550 raft_consensus.cc:348] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.109817 29550 raft_consensus.cc:374] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:49.109925 29550 raft_consensus.cc:729] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6115409b66e42ce8731c180d1ece2cc, State: Initialized, Role: FOLLOWER
I20260810 23:42:49.110692 29550 consensus_queue.cc:260] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.111373 29552 raft_consensus.cc:348] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.111922 29552 raft_consensus.cc:374] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:49.112032 29552 raft_consensus.cc:729] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9189b1c8a5ac4ab4814339d97153803f, State: Initialized, Role: FOLLOWER
I20260810 23:42:49.112048 29263 heartbeater.cc:503] Master 127.30.33.62:36009 was elected leader, sending a full tablet report...
I20260810 23:42:49.112643 29552 consensus_queue.cc:260] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.112828 29550 ts_tablet_manager.cc:1428] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260810 23:42:49.113842 29521 heartbeater.cc:503] Master 127.30.33.62:36009 was elected leader, sending a full tablet report...
I20260810 23:42:49.114297 29552 ts_tablet_manager.cc:1428] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 23:42:49.116990 29551 raft_consensus.cc:348] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.117520 29551 raft_consensus.cc:374] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:42:49.117650 29551 raft_consensus.cc:729] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5a2ce46b444e07acd9b9096c47e084, State: Initialized, Role: FOLLOWER
I20260810 23:42:49.118332 29551 consensus_queue.cc:260] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.120342 29551 ts_tablet_manager.cc:1428] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260810 23:42:49.209760 29524 tablet.cc:2369] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:42:49.216605 29557 raft_consensus.cc:482] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:42:49.217054 29557 raft_consensus.cc:504] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.218936 29557 leader_election.cc:288] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507), 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165)
I20260810 23:42:49.226447 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.219617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:56333 (local address 127.30.33.1:34165)
0810 23:42:49.219821 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.219827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.220833 (+ 1006us) server_negotiation.cc:407] Connection header received
0810 23:42:49.221085 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.221090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.221173 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.221256 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:49.221915 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.222764 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.224673 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.225459 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.225588 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.225709 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.226068 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.226206 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:42:49.226524 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.219404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:36755 (local address 127.30.33.3:43507)
0810 23:42:49.219696 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.219703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.220808 (+ 1105us) server_negotiation.cc:407] Connection header received
0810 23:42:49.221053 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.221058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.221165 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.221267 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:42:49.222146 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.222991 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.224791 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.225588 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.225717 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.225822 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.226140 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.226280 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 23:42:49.226626 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.219335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.30.33.2:56333)
0810 23:42:49.220736 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.220751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:49.220917 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:49.221292 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:49.221297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:49.221327 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:49.221749 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.221762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.222908 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.222913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.224488 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.224504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.225700 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.225707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.225905 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.225935 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:42:49.226022 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:42:49.226169 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"mutex_wait_us":212,"thread_start_us":500,"threads_started":1}
I20260810 23:42:49.226671 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.219335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43507 (local address 127.30.33.2:36755)
0810 23:42:49.220679 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.220700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:42:49.220891 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:42:49.221457 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:42:49.221462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:42:49.221493 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:42:49.221977 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.221988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.223124 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.223129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.224628 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:42:49.224643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.225807 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.225814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:42:49.225986 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.226010 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:42:49.226109 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:42:49.226248 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":3,"thread_start_us":534,"threads_started":1}
I20260810 23:42:49.227680 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034f34fb8e724ec29739dade39b9d197" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9189b1c8a5ac4ab4814339d97153803f" is_pre_election: true
I20260810 23:42:49.228019 28931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034f34fb8e724ec29739dade39b9d197" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5a2ce46b444e07acd9b9096c47e084" is_pre_election: true
I20260810 23:42:49.228336 29441 raft_consensus.cc:2454] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 0.
I20260810 23:42:49.228595 28931 raft_consensus.cc:2454] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 0.
I20260810 23:42:49.229197 29052 leader_election.cc:302] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9189b1c8a5ac4ab4814339d97153803f, d6115409b66e42ce8731c180d1ece2cc; no voters:
I20260810 23:42:49.229633 29557 raft_consensus.cc:2781] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:42:49.229790 29557 raft_consensus.cc:482] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:42:49.229883 29557 raft_consensus.cc:3037] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:42:49.230315 29264 tablet.cc:2369] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:42:49.232376 29557 raft_consensus.cc:504] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
I20260810 23:42:49.233050 29557 leader_election.cc:288] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 election: Requested vote from peers 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507), 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165)
I20260810 23:42:49.233610 28931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034f34fb8e724ec29739dade39b9d197" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5a2ce46b444e07acd9b9096c47e084"
I20260810 23:42:49.233641 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034f34fb8e724ec29739dade39b9d197" candidate_uuid: "d6115409b66e42ce8731c180d1ece2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9189b1c8a5ac4ab4814339d97153803f"
I20260810 23:42:49.233827 28931 raft_consensus.cc:3037] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:49.233863 29441 raft_consensus.cc:3037] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:42:49.236442 28931 raft_consensus.cc:2454] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 1.
I20260810 23:42:49.236441 29441 raft_consensus.cc:2454] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6115409b66e42ce8731c180d1ece2cc in term 1.
I20260810 23:42:49.236931 29052 leader_election.cc:302] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9189b1c8a5ac4ab4814339d97153803f, d6115409b66e42ce8731c180d1ece2cc; no voters:
I20260810 23:42:49.237233 29557 raft_consensus.cc:2781] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:42:49.237694 29557 raft_consensus.cc:686] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 1 LEADER]: Becoming Leader. State: Replica: d6115409b66e42ce8731c180d1ece2cc, State: Running, Role: LEADER
I20260810 23:42:49.238183 29557 consensus_queue.cc:237] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } }
W20260810 23:42:49.241531 29009 tablet.cc:2369] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:42:49.243340 28667 catalog_manager.cc:5495] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc reported cstate change: term changed from 0 to 1, leader changed from <none> to d6115409b66e42ce8731c180d1ece2cc (127.30.33.2). New cstate: current_term: 1 leader_uuid: "d6115409b66e42ce8731c180d1ece2cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6115409b66e42ce8731c180d1ece2cc" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 } health_report { overall_health: UNKNOWN } } }
I20260810 23:42:49.271227 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.265566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.30.33.2:33001)
0810 23:42:49.265803 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.265808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.265826 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:42:49.266013 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.266017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.266112 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.266242 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:42:49.266676 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.267429 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.268924 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.269376 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.269474 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.270062 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:42:49.270616 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:42:49.270741 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.270880 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.270955 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:42:49.279913 28931 raft_consensus.cc:1261] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Refusing update from remote peer d6115409b66e42ce8731c180d1ece2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:49.279915 29441 raft_consensus.cc:1261] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Refusing update from remote peer d6115409b66e42ce8731c180d1ece2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:42:49.280936 29566 consensus_queue.cc:1035] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9189b1c8a5ac4ab4814339d97153803f" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:49.281304 29557 consensus_queue.cc:1035] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5a2ce46b444e07acd9b9096c47e084" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:42:49.295357 29572 mvcc.cc:205] Tried to move back new op lower bound from 7317116392559722496 to 7317116392403709952. Current Snapshot: MvccSnapshot[applied={T|T < 7317116392559722496}]
I20260810 23:42:49.310379 28667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:49.310636 28667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:42:49.314488 28667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Sending DeleteTablet for 3 replicas of tablet 034f34fb8e724ec29739dade39b9d197
I20260810 23:42:49.315863 29405 tablet_service.cc:1514] Processing DeleteTablet for tablet 034f34fb8e724ec29739dade39b9d197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:49 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:54958
I20260810 23:42:49.315869 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 034f34fb8e724ec29739dade39b9d197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:49 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:34920
I20260810 23:42:49.316126 28891 tablet_service.cc:1514] Processing DeleteTablet for tablet 034f34fb8e724ec29739dade39b9d197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:42:49 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:45516
I20260810 23:42:49.317283 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 28734
I20260810 23:42:49.334290 29592 tablet_replica.cc:330] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: stopping tablet replica
I20260810 23:42:49.334903 29592 raft_consensus.cc:2229] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:49.335613 29592 raft_consensus.cc:2258] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:49.338163 29592 ts_tablet_manager.cc:1905] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:49.342762 29589 tablet_replica.cc:330] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: stopping tablet replica
I20260810 23:42:49.343439 29589 raft_consensus.cc:2229] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:42:49.344276 29589 raft_consensus.cc:2258] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:49.344722 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.318994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.30.33.1:34165)
0810 23:42:49.319297 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.319303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.319325 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:42:49.319393 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.319399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.319514 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.319619 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:49.321769 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.323127 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.325411 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.326197 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.326355 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.334639 (+ 8284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:49.334664 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:49.334686 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:49.334744 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:49.339225 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.340464 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.340476 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.340480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.340710 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.341237 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.341243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.341246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.341805 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:49.342175 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.343687 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.344142 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260810 23:42:49.345618 28998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:49.345965 29588 tablet_replica.cc:330] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: stopping tablet replica
I20260810 23:42:49.346493 29588 raft_consensus.cc:2229] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:42:49.347081 29588 raft_consensus.cc:2258] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:42:49.347311 29589 ts_tablet_manager.cc:1905] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:49.349877 29588 ts_tablet_manager.cc:1905] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:42:49.355779 29589 ts_tablet_manager.cc:1918] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:49.356091 29589 log.cc:1196] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/034f34fb8e724ec29739dade39b9d197
I20260810 23:42:49.356812 29589 ts_tablet_manager.cc:1939] T 034f34fb8e724ec29739dade39b9d197 P d6115409b66e42ce8731c180d1ece2cc: Deleting consensus metadata
I20260810 23:42:49.357842 29588 ts_tablet_manager.cc:1918] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:49.358039 29588 log.cc:1196] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/034f34fb8e724ec29739dade39b9d197
I20260810 23:42:49.358625 29588 ts_tablet_manager.cc:1939] T 034f34fb8e724ec29739dade39b9d197 P 9189b1c8a5ac4ab4814339d97153803f: Deleting consensus metadata
I20260810 23:42:49.359233 28645 catalog_manager.cc:4841] TS d6115409b66e42ce8731c180d1ece2cc (127.30.33.2:33001): tablet 034f34fb8e724ec29739dade39b9d197 (table test-table [id=096604a6c8da446ba50a8f5f0a17df1e]) successfully deleted
I20260810 23:42:49.360628 28645 catalog_manager.cc:4841] TS 9189b1c8a5ac4ab4814339d97153803f (127.30.33.3:43507): tablet 034f34fb8e724ec29739dade39b9d197 (table test-table [id=096604a6c8da446ba50a8f5f0a17df1e]) successfully deleted
I20260810 23:42:49.451144 29592 ts_tablet_manager.cc:1918] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:42:49.451447 29592 log.cc:1196] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/034f34fb8e724ec29739dade39b9d197
I20260810 23:42:49.452026 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 28734
I20260810 23:42:49.452042 29592 ts_tablet_manager.cc:1939] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting consensus metadata
W20260810 23:42:49.485998 28647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.33.1:34165 (error 108)
I20260810 23:42:49.486439 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 29014
I20260810 23:42:49.489003 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.488502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45534)
0810 23:42:49.488706 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.488836 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 23:42:49.489288 28647 catalog_manager.cc:4584] TS 3a5a2ce46b444e07acd9b9096c47e084 (127.30.33.1:34165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 034f34fb8e724ec29739dade39b9d197: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
I20260810 23:42:49.510301 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.487276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.30.33.2:33001)
0810 23:42:49.487864 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.487870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.487890 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:42:49.487955 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.487960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.488045 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.488143 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:49.490012 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.491350 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.493324 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.494088 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.494239 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.502629 (+ 8390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:49.502655 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:49.502675 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:49.502728 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:49.507043 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.507974 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.507988 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.507992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.508182 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.508576 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.508581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.508585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.509097 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:49.509438 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.509634 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.509887 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448}
I20260810 23:42:49.511042 29256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:49.543180 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.542531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45536)
0810 23:42:49.542855 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.542983 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260810 23:42:49.544404 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.544020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45542)
0810 23:42:49.544180 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.544266 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:42:49.605361 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.604808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45550)
0810 23:42:49.605054 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.605157 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:42:49.606487 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.606077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45552)
0810 23:42:49.606262 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.606351 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:42:49.628266 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 29014
I20260810 23:42:49.653811 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 29267
I20260810 23:42:49.668169 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.654724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.30.33.3:43507)
0810 23:42:49.654931 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:42:49.654937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:42:49.654960 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:42:49.655161 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:42:49.655166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:42:49.655233 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:42:49.655325 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:42:49.656770 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.657570 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.658956 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:42:49.659375 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:42:49.659688 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:42:49.662420 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:42:49.662439 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:42:49.662456 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:42:49.662498 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:42:49.665231 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.666139 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.666147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.666150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.666304 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:42:49.666701 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:42:49.666705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:42:49.666707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:42:49.667118 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:42:49.667348 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 23:42:49.667696 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 23:42:49.667883 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:42:49.668871 29513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:49.687484 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.686852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45560)
0810 23:42:49.687138 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.687246 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 23:42:49.688843 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.688288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45564)
0810 23:42:49.688585 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.688703 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260810 23:42:49.782366 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 29267
I20260810 23:42:49.802160 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 28608
I20260810 23:42:49.803503 28718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:42:49.920490 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 28608
I20260810 23:42:49.920615 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:42:49.920042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34165 (local address 127.0.0.1:45578)
0810 23:42:49.920295 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 23:42:49.920409 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 23:43:09.951712 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36009
--webserver_interface=127.30.33.62
--webserver_port=41751
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:10.304138 29608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:10.304491 29608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:10.304646 29608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:10.312851 29608 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:43:10.312992 29608 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:43:10.313058 29608 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:43:10.313127 29608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:10.313205 29608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:10.313303 29608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:10.313366 29608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:10.313441 29608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:10.323997 29608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:10.325400 29608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:10.326853 29608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:10.339354 29614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:10.339387 29623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:10.339443 29617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:10.339731 29608 server_base.cc:1034] running on GCE node
I20260810 23:43:10.340905 29608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:10.342401 29608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:10.343583 29608 hybrid_clock.cc:648] HybridClock initialized: now 1786405390343495 us; error 59 us; skew 500 ppm
I20260810 23:43:10.343958 29608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:10 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405390, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:10.347318 29608 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:43:10.348874 29608 webserver.cc:460] Webserver started at http://127.30.33.62:41751/ using document root <none> and password file <none>
I20260810 23:43:10.349628 29608 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:10.349746 29608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:10.355734 29608 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:43:10.358819 29632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:10.360363 29608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:10.360543 29608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "9d31291a6aec436db87c30f3fb15f59b"
format_stamp: "Formatted at 2026-08-10 23:42:23 on dist-test-slave-pww0"
server_key: "bc15556cd5023f023deef70e0f3da139"
server_key_iv: "910405615564914e68c9ea54838158be"
server_key_version: "encryptionkey@0"
I20260810 23:43:10.361534 29608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:10.388175 29608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:10.393054 29608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:10.393383 29608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:10.412247 29608 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:36009
I20260810 23:43:10.412272 29736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:36009 every 8 connection(s)
I20260810 23:43:10.413834 29608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:10.414918 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 29608
I20260810 23:43:10.415882 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:34165
--local_ip_for_outbound_sockets=127.30.33.1
--tserver_master_addrs=127.30.33.62:36009
--webserver_port=46805
--webserver_interface=127.30.33.1
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:43:10.421521 29737 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:43:10.425084 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Bootstrap starting.
I20260810 23:43:10.446933 29737 log.cc:824] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:10.464685 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:43:10.465341 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Bootstrap complete.
I20260810 23:43:10.475050 29737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:43:10.476125 29737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d31291a6aec436db87c30f3fb15f59b, State: Initialized, Role: FOLLOWER
I20260810 23:43:10.476884 29737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:43:10.477116 29737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:10.477245 29737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:10.477415 29737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:43:10.481585 29737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:43:10.482152 29737 leader_election.cc:302] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d31291a6aec436db87c30f3fb15f59b; no voters:
I20260810 23:43:10.482831 29737 leader_election.cc:288] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:43:10.482982 29743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:43:10.485049 29743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [term 3 LEADER]: Becoming Leader. State: Replica: 9d31291a6aec436db87c30f3fb15f59b, State: Running, Role: LEADER
I20260810 23:43:10.485257 29737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:10.485657 29743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } }
I20260810 23:43:10.489045 29745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d31291a6aec436db87c30f3fb15f59b. Latest consensus state: current_term: 3 leader_uuid: "9d31291a6aec436db87c30f3fb15f59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } } }
I20260810 23:43:10.489466 29745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:10.490031 29744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9d31291a6aec436db87c30f3fb15f59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d31291a6aec436db87c30f3fb15f59b" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36009 } } }
I20260810 23:43:10.490387 29744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:10.501533 29758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:10.511865 29758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:10.513846 29758 catalog_manager.cc:1259] Loaded cluster ID: 7add57300a114fc886eecd1763a8ebc3
I20260810 23:43:10.514441 29758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:10.519389 29758 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:10.521553 29758 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Loaded TSK: 0
I20260810 23:43:10.521844 29758 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d31291a6aec436db87c30f3fb15f59b: Loaded TSK: 1
F20260810 23:43:10.523367 29758 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd9a09eb9ad google::LogMessage::Fail() at ??:0
@ 0x7fd9a09ee88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd9a09eb4de google::LogMessage::Flush() at ??:0
@ 0x7fd9a09eca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd9a474f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd9a473e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd9b641e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd9b641d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd9b64a92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd9b64a90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd9b64f35e9 std::function<>::operator()() at ??:0
@ 0x7fd9b6421cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd9b641fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd9b64a7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd9b64a7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd9a3637345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd9a363a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd9a363a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd9a3615eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd9a68ef6db start_thread at ??:0
@ 0x7fd99f88561f clone at ??:0
W20260810 23:43:10.825306 29741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:10.825644 29741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:10.825790 29741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:10.833666 29741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:10.833827 29741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:10.833997 29741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:43:10.844566 29741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:34165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=46805
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:10.845980 29741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:10.847520 29741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:10.861374 29824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:10.861397 29826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:10.861332 29829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:10.862180 29741 server_base.cc:1034] running on GCE node
I20260810 23:43:10.862910 29741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:10.864024 29741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:10.865243 29741 hybrid_clock.cc:648] HybridClock initialized: now 1786405390865179 us; error 64 us; skew 500 ppm
I20260810 23:43:10.865582 29741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:10 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405390, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:10.868937 29741 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:43:10.870622 29741 webserver.cc:460] Webserver started at http://127.30.33.1:46805/ using document root <none> and password file <none>
I20260810 23:43:10.871402 29741 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:10.871578 29741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:10.877786 29741 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 23:43:10.881260 29841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:10.882953 29741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:43:10.883144 29741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "3a5a2ce46b444e07acd9b9096c47e084"
format_stamp: "Formatted at 2026-08-10 23:42:24 on dist-test-slave-pww0"
server_key: "bdeabde505ed412e2efb187dbdb51143"
server_key_iv: "ab5a1352dd98aa362151170326ccd128"
server_key_version: "encryptionkey@0"
I20260810 23:43:10.884296 29741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:10.910132 29741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:10.915059 29741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:10.915398 29741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:10.916611 29741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:10.918924 29862 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260810 23:43:10.925009 29862 ts_tablet_manager.cc:1852] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 23:43:10.926098 29862 ts_tablet_manager.cc:1905] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:10.934875 29862 ts_tablet_manager.cc:1918] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:10.935240 29862 ts_tablet_manager.cc:1939] T 034f34fb8e724ec29739dade39b9d197 P 3a5a2ce46b444e07acd9b9096c47e084: Deleting consensus metadata
I20260810 23:43:10.935870 29741 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 23:43:10.936003 29741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.000s
I20260810 23:43:10.936117 29741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:10.936179 29741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:10.973083 29741 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:34165
I20260810 23:43:10.973146 30068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:34165 every 8 connection(s)
I20260810 23:43:10.974498 29741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:43:10.979959 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:10.977841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:57361)
0810 23:43:10.978693 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:10.978948 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":341,"threads_started":1}
W20260810 23:43:10.980687 30069 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:36009 (0 consecutive failures): Network error: Failed to ping master at 127.30.33.62:36009: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
I20260810 23:43:10.980901 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 29741
I20260810 23:43:10.981693 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:33001
--local_ip_for_outbound_sockets=127.30.33.2
--tserver_master_addrs=127.30.33.62:36009
--webserver_port=37563
--webserver_interface=127.30.33.2
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:43:10.981779 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:10.981357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:39865)
0810 23:43:10.981545 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:10.981652 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 23:43:10.983022 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:10.982617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:43931)
0810 23:43:10.982783 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:10.982884 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 23:43:10.983261 30069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:43:11.341867 30074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:11.342186 30074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:11.342286 30074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:11.349852 30074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:11.349983 30074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:11.350102 30074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:43:11.360333 30074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:33001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=37563
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:11.361689 30074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:11.363099 30074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:11.377408 30083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:11.377456 30082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:11.377367 30086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:11.378587 30074 server_base.cc:1034] running on GCE node
I20260810 23:43:11.379298 30074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:11.380460 30074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:11.381675 30074 hybrid_clock.cc:648] HybridClock initialized: now 1786405391381613 us; error 61 us; skew 500 ppm
I20260810 23:43:11.382004 30074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:11 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405391, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:11.385358 30074 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:43:11.386986 30074 webserver.cc:460] Webserver started at http://127.30.33.2:37563/ using document root <none> and password file <none>
I20260810 23:43:11.387815 30074 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:11.387953 30074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:11.394129 30074 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:43:11.397284 30104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:11.398731 30074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:43:11.398917 30074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "d6115409b66e42ce8731c180d1ece2cc"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "22572741899e2a110ed0b6b010533375"
server_key_iv: "e1e7d2a9e3cb689329bfa18997a13f70"
server_key_version: "encryptionkey@0"
I20260810 23:43:11.400070 30074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:11.423372 30074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:11.428139 30074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:11.428442 30074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:11.429572 30074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:11.431111 30074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:11.431216 30074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:11.431346 30074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:11.431418 30074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:11.464110 30074 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:33001
I20260810 23:43:11.464205 30332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:33001 every 8 connection(s)
I20260810 23:43:11.465624 30074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:43:11.470882 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:11.468779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:57499)
0810 23:43:11.469623 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:11.469873 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":334,"threads_started":1}
W20260810 23:43:11.471756 30337 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:36009 (0 consecutive failures): Network error: Failed to ping master at 127.30.33.62:36009: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
I20260810 23:43:11.472754 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:11.472327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:38597)
0810 23:43:11.472513 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:11.472597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:43:11.473600 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:11.473288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:58587)
0810 23:43:11.473406 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:11.473483 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:43:11.473861 30337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:43:11.475659 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 30074
I20260810 23:43:11.476313 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:43507
--local_ip_for_outbound_sockets=127.30.33.3
--tserver_master_addrs=127.30.33.62:36009
--webserver_port=37091
--webserver_interface=127.30.33.3
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:11.829268 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:11.829583 30340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:11.829726 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:11.837216 30340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:11.837354 30340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:11.837517 30340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:43:11.847626 30340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:43507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=37091
--tserver_master_addrs=127.30.33.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:11.849050 30340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:11.850476 30340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:11.863767 30347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:11.863816 30351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:11.864367 30345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:11.864656 30340 server_base.cc:1034] running on GCE node
I20260810 23:43:11.865243 30340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:11.866245 30340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:11.867483 30340 hybrid_clock.cc:648] HybridClock initialized: now 1786405391867405 us; error 43 us; skew 500 ppm
I20260810 23:43:11.867841 30340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:11 dist-test-slave-pww0 krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405391, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:11.871028 30340 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:43:11.872548 30340 webserver.cc:460] Webserver started at http://127.30.33.3:37091/ using document root <none> and password file <none>
I20260810 23:43:11.873251 30340 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:11.873373 30340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:11.879007 30340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 23:43:11.881934 30364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:11.883360 30340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:43:11.883574 30340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "9189b1c8a5ac4ab4814339d97153803f"
format_stamp: "Formatted at 2026-08-10 23:42:25 on dist-test-slave-pww0"
server_key: "46b52db37283b48f11c465dcf9c2f57f"
server_key_iv: "62f2d444687521326436f9574c8091dd"
server_key_version: "encryptionkey@0"
I20260810 23:43:11.884625 30340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:11.911980 30340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:11.916441 30340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:11.916738 30340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:11.917847 30340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:11.919292 30340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:11.919382 30340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:11.919503 30340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:11.919584 30340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:11.952266 30340 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:43507
I20260810 23:43:11.952356 30561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:43507 every 8 connection(s)
I20260810 23:43:11.954276 30340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:43:11.959434 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:11.957422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:41731)
0810 23:43:11.958209 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:11.958490 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":314,"threads_started":1}
W20260810 23:43:11.960235 30562 heartbeater.cc:644] Failed to heartbeat to 127.30.33.62:36009 (0 consecutive failures): Network error: Failed to ping master at 127.30.33.62:36009: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
I20260810 23:43:11.961050 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 30340
I20260810 23:43:11.961108 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:11.960762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:58367)
0810 23:43:11.960922 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:11.960994 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:43:11.962112 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:11.961713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:36739)
0810 23:43:11.961875 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:11.961964 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 23:43:11.962435 30562 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:43:11.985090 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:11.984116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:55833)
0810 23:43:11.984679 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:11.984776 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":259,"threads_started":1}
I20260810 23:43:12.475521 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:12.474477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:46487)
0810 23:43:12.475056 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:12.475185 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":293,"threads_started":1}
I20260810 23:43:12.964033 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:12.963131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:33353)
0810 23:43:12.963675 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:12.963781 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":239,"threads_started":1}
I20260810 23:43:12.987025 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:12.986215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:38451)
0810 23:43:12.986730 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:12.986838 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":261,"threads_started":1}
I20260810 23:43:13.477447 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:13.476635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:58063)
0810 23:43:13.477159 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:13.477260 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":256,"threads_started":1}
I20260810 23:43:13.965909 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:13.965046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:53943)
0810 23:43:13.965607 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:13.965726 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":276,"threads_started":1}
I20260810 23:43:13.989056 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:13.988132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:37711)
0810 23:43:13.988744 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:13.988850 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":318,"threads_started":1}
I20260810 23:43:14.479563 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:14.478548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:45513)
0810 23:43:14.479183 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:14.479299 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":325,"threads_started":1}
I20260810 23:43:14.967954 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:14.967037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:55459)
0810 23:43:14.967661 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:14.967757 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":321,"threads_started":1}
I20260810 23:43:14.990936 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:14.990096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:48189)
0810 23:43:14.990626 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:14.990734 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":296,"threads_started":1}
I20260810 23:43:15.481380 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:15.480609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:54665)
0810 23:43:15.481089 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:15.481191 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":234,"threads_started":1}
I20260810 23:43:15.969769 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:15.968940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:35809)
0810 23:43:15.969463 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:15.969558 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":259,"threads_started":1}
I20260810 23:43:15.992820 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:15.992008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:40283)
0810 23:43:15.992529 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:15.992624 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
I20260810 23:43:16.483282 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:16.482458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:43555)
0810 23:43:16.482995 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:16.483083 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
I20260810 23:43:16.971662 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:16.970736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:56749)
0810 23:43:16.971316 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:16.971409 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
I20260810 23:43:16.994577 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:16.993762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:38945)
0810 23:43:16.994303 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:16.994390 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":287,"threads_started":1}
I20260810 23:43:17.484983 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:17.484230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:35173)
0810 23:43:17.484703 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:17.484811 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":238,"threads_started":1}
I20260810 23:43:17.973935 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:17.972709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:34469)
0810 23:43:17.973662 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:17.973769 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":711,"threads_started":1}
I20260810 23:43:17.996562 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:17.995682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:43751)
0810 23:43:17.996267 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:17.996353 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":256,"threads_started":1}
I20260810 23:43:18.486776 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:18.485923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:46839)
0810 23:43:18.486439 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:18.486527 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":222,"threads_started":1}
I20260810 23:43:18.976132 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:18.975026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:53825)
0810 23:43:18.975717 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:18.975838 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":288,"threads_started":1}
I20260810 23:43:18.998386 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:18.997560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:58527)
0810 23:43:18.998082 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:18.998185 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
I20260810 23:43:19.488531 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:19.487767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:32851)
0810 23:43:19.488278 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:19.488361 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":268,"threads_started":1}
I20260810 23:43:19.978291 30619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:19.977429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:42827)
0810 23:43:19.978005 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:19.978119 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":295,"threads_started":1}
I20260810 23:43:20.000159 30621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:19.999374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:45519)
0810 23:43:19.999888 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:19.999985 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":261,"threads_started":1}
I20260810 23:43:20.490356 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:20.489546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:51925)
0810 23:43:20.490059 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:20.490145 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":250,"threads_started":1}
I20260810 23:43:20.980262 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:20.979345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:39131)
0810 23:43:20.979902 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:20.979998 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":288,"threads_started":1}
I20260810 23:43:21.001964 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:21.001208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:39401)
0810 23:43:21.001666 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:21.001766 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":233,"threads_started":1}
I20260810 23:43:21.492131 30628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:21.491360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:45783)
0810 23:43:21.491853 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:21.491949 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":264,"threads_started":1}
I20260810 23:43:21.982299 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:21.981463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:58857)
0810 23:43:21.982014 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:21.982112 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":307,"threads_started":1}
I20260810 23:43:22.003974 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:22.003040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:41339)
0810 23:43:22.003660 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:22.003781 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":308,"threads_started":1}
I20260810 23:43:22.493929 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:22.493171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:51317)
0810 23:43:22.493621 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:22.493738 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260810 23:43:22.984143 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:22.983363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:47395)
0810 23:43:22.983852 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:22.983959 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
I20260810 23:43:23.005793 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:23.005010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:51057)
0810 23:43:23.005514 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:23.005608 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260810 23:43:23.495795 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:23.494988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:47061)
0810 23:43:23.495520 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:23.495616 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260810 23:43:23.985976 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:23.985195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:60455)
0810 23:43:23.985717 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:23.985810 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
I20260810 23:43:24.007544 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:24.006721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:36405)
0810 23:43:24.007221 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:24.007332 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":250,"threads_started":1}
I20260810 23:43:24.497822 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:24.496931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:41413)
0810 23:43:24.497492 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:24.497607 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":291,"threads_started":1}
I20260810 23:43:24.987955 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:24.986970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:57237)
0810 23:43:24.987522 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:24.987695 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260810 23:43:25.009286 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:25.008473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:44409)
0810 23:43:25.008989 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:25.009081 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":262,"threads_started":1}
I20260810 23:43:25.499655 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:25.498854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:60465)
0810 23:43:25.499327 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:25.499414 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
I20260810 23:43:25.989725 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:25.988934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:43035)
0810 23:43:25.989415 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:25.989508 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260810 23:43:26.011256 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:26.010444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.1:50907)
0810 23:43:26.010967 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:26.011084 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":232,"threads_started":1}
I20260810 23:43:26.501760 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:26.500920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.2:34519)
0810 23:43:26.501425 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:26.501553 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260810 23:43:26.970597 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 29741
I20260810 23:43:26.991775 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:26.990908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36009 (local address 127.30.33.3:42715)
0810 23:43:26.991413 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:26.991574 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260810 23:43:26.996433 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:26.971903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.30.33.1:34165)
0810 23:43:26.972502 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:26.972523 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:26.972576 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:43:26.974365 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:26.974377 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:26.974890 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:26.975283 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:26.977029 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:26.978413 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:26.979998 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:26.980786 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:26.980992 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:26.988537 (+ 7545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:26.988549 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:26.988576 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:26.988644 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:26.992426 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:26.993309 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:26.993319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:26.993327 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:26.993538 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:26.993985 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:26.993990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:26.993992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:26.994639 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:26.994978 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:26.995576 (+ 598us) server_negotiation.cc:299] Negotiation successful
0810 23:43:26.996092 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":756,"threads_started":1}
I20260810 23:43:26.998026 30062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:27.087101 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 29741
I20260810 23:43:27.116796 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 30074
I20260810 23:43:27.138255 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:27.117796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.30.33.2:33001)
0810 23:43:27.118361 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:27.118376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:27.118417 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:43:27.120086 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:27.120098 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:27.120587 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:27.120962 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:27.122527 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:27.123950 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:27.125394 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:27.126171 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:27.126366 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:27.130134 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:27.130149 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:27.130174 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:27.130236 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:27.134390 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:27.135244 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:27.135255 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:27.135264 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:27.135507 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:27.135967 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:27.135972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:27.135976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:27.136598 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:27.136903 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:27.137465 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 23:43:27.137936 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":252,"threads_started":1}
I20260810 23:43:27.139900 30327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:27.225193 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 30074
I20260810 23:43:27.251905 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 30340
I20260810 23:43:27.271832 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:27.252887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.30.33.3:43507)
0810 23:43:27.253578 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:27.253593 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:27.253637 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:43:27.255375 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:27.255387 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:27.255920 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:27.256275 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:27.257778 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:27.259095 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:27.260602 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:27.261400 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:27.261552 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:27.264501 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:27.264510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:27.264528 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:27.264576 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:27.267958 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:27.268906 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:27.268915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:27.268924 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:27.269151 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:27.269576 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:27.269579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:27.269581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:27.270214 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:27.270491 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:27.270960 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 23:43:27.271451 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":327,"threads_started":1}
I20260810 23:43:27.273389 30554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:27.359449 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 30340
2026-08-10T23:43:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65480 ms)
[----------] 30 tests from SecurityITest (316273 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:43:27 dist-test-slave-pww0 krb5kdc[30677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:43:27 dist-test-slave-pww0 krb5kdc[30677](info): set up 2 sockets
Aug 10 23:43:27 dist-test-slave-pww0 krb5kdc[30677](info): commencing operation
krb5kdc: starting...
W20260810 23:43:28.684736 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.005s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:43:28 dist-test-slave-pww0 krb5kdc[30677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:43:29.023049 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:43615
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:29.386725 30701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:29.387038 30701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:29.387164 30701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:29.394654 30701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:29.394781 30701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:29.394836 30701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:29.394917 30701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:29.394968 30701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:29.405364 30701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:43615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:29.406778 30701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:29.408262 30701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:29.420691 30708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:29.420725 30709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:29.420667 30711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:29.421262 30701 server_base.cc:1034] running on GCE node
I20260810 23:43:29.422220 30701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:29.423930 30701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:29.425179 30701 hybrid_clock.cc:648] HybridClock initialized: now 1786405409425114 us; error 52 us; skew 500 ppm
I20260810 23:43:29.425628 30701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:29 dist-test-slave-pww0 krb5kdc[30677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405409, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:29.428923 30701 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:43:29.430531 30701 webserver.cc:460] Webserver started at http://127.30.33.62:35601/ using document root <none> and password file <none>
I20260810 23:43:29.431267 30701 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:29.431394 30701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:29.431730 30701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:29.434890 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "79650e3be93f41e69e6d3ddd57564d0a"
format_stamp: "Formatted at 2026-08-10 23:43:29 on dist-test-slave-pww0"
server_key: "005584eeab9e5d1306e7c0c9352b531f"
server_key_iv: "586f9ee630c0fb55ce6007b674fc2908"
server_key_version: "encryptionkey@0"
I20260810 23:43:29.435766 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "79650e3be93f41e69e6d3ddd57564d0a"
format_stamp: "Formatted at 2026-08-10 23:43:29 on dist-test-slave-pww0"
server_key: "005584eeab9e5d1306e7c0c9352b531f"
server_key_iv: "586f9ee630c0fb55ce6007b674fc2908"
server_key_version: "encryptionkey@0"
I20260810 23:43:29.441435 30701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 23:43:29.445377 30719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:29.446853 30701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:43:29.447041 30701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "79650e3be93f41e69e6d3ddd57564d0a"
format_stamp: "Formatted at 2026-08-10 23:43:29 on dist-test-slave-pww0"
server_key: "005584eeab9e5d1306e7c0c9352b531f"
server_key_iv: "586f9ee630c0fb55ce6007b674fc2908"
server_key_version: "encryptionkey@0"
I20260810 23:43:29.447250 30701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:29.486163 30701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:29.490795 30701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:29.491082 30701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:29.510083 30701 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:43615
I20260810 23:43:29.510090 30833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:43615 every 8 connection(s)
I20260810 23:43:29.511662 30701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:29.515623 30835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:29.517803 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 30701
I20260810 23:43:29.518008 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:29.518687 30852 external_mini_cluster.cc:1428] Setting key 2a7faec481b477392ccdeae31f017935
I20260810 23:43:29.529193 30835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a: Bootstrap starting.
I20260810 23:43:29.533121 30835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:29.534309 30835 log.cc:824] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:43:29 dist-test-slave-pww0 krb5kdc[30677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:29.537791 30835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a: No bootstrap required, opened a new log
I20260810 23:43:29.546662 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:29.521150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.30.33.62:43615)
0810 23:43:29.522033 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:29.522051 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:29.522106 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:43:29.524596 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:29.524631 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:29.525252 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:29.525798 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:29.527333 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:29.529443 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:29.531065 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:29.532166 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:29.532374 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:29.537216 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:29.537229 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:29.537252 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:29.537302 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:29.541441 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:29.542270 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:29.542279 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:29.542290 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:29.542539 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:29.543051 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:29.543056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:29.543060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:29.543903 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:29.544210 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:29.544651 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 23:43:29.545153 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":366,"threads_started":1}
I20260810 23:43:29.548525 30835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79650e3be93f41e69e6d3ddd57564d0a" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43615 } }
I20260810 23:43:29.549021 30835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:29.549149 30835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79650e3be93f41e69e6d3ddd57564d0a, State: Initialized, Role: FOLLOWER
I20260810 23:43:29.549816 30835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79650e3be93f41e69e6d3ddd57564d0a" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43615 } }
I20260810 23:43:29.550086 30835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:29.550218 30835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:29.550401 30835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:29.552750 30835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79650e3be93f41e69e6d3ddd57564d0a" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43615 } }
I20260810 23:43:29.553289 30835 leader_election.cc:302] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79650e3be93f41e69e6d3ddd57564d0a; no voters:
I20260810 23:43:29.553898 30835 leader_election.cc:288] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:29.554035 30841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:29.554749 30841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [term 1 LEADER]: Becoming Leader. State: Replica: 79650e3be93f41e69e6d3ddd57564d0a, State: Running, Role: LEADER
I20260810 23:43:29.555310 30841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79650e3be93f41e69e6d3ddd57564d0a" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43615 } }
I20260810 23:43:29.556198 30835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:29.558507 30842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79650e3be93f41e69e6d3ddd57564d0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79650e3be93f41e69e6d3ddd57564d0a" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43615 } } }
I20260810 23:43:29.559000 30842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:29.558517 30844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79650e3be93f41e69e6d3ddd57564d0a. Latest consensus state: current_term: 1 leader_uuid: "79650e3be93f41e69e6d3ddd57564d0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79650e3be93f41e69e6d3ddd57564d0a" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 43615 } } }
I20260810 23:43:29.559600 30844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:29.560611 30857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:29.566129 30857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:29.576037 30857 catalog_manager.cc:1347] Generated new cluster ID: d7cb1c5fdfde4073860f0eb4a2578138
I20260810 23:43:29.576182 30857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:29.595578 30857 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:29.597687 30857 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:29.608006 30857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a: Generated new TSK 0
I20260810 23:43:29.608939 30857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:43:29.693250 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43615
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:30.079304 30878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:30.079720 30878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:30.079891 30878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:30.087613 30878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:30.087777 30878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:30.087956 30878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:43:30.098433 30878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:30.099987 30878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:30.101373 30878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:30.114989 30890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:30.115006 30886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:30.115056 30885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:30.121272 30878 server_base.cc:1034] running on GCE node
I20260810 23:43:30.122035 30878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:30.123114 30878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:30.124336 30878 hybrid_clock.cc:648] HybridClock initialized: now 1786405410124257 us; error 82 us; skew 500 ppm
I20260810 23:43:30.124707 30878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:30 dist-test-slave-pww0 krb5kdc[30677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405410, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:30.128170 30878 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:43:30.129908 30878 webserver.cc:460] Webserver started at http://127.30.33.1:40975/ using document root <none> and password file <none>
I20260810 23:43:30.130714 30878 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:30.130841 30878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:30.131160 30878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:30.134287 30878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "bb6b68019e1546aab67364314e7044ca"
format_stamp: "Formatted at 2026-08-10 23:43:30 on dist-test-slave-pww0"
server_key: "47df4c8e517128ba36717c58b8a987a7"
server_key_iv: "28c72ddd11ed85f4b61e3313816176ef"
server_key_version: "encryptionkey@0"
I20260810 23:43:30.135089 30878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "bb6b68019e1546aab67364314e7044ca"
format_stamp: "Formatted at 2026-08-10 23:43:30 on dist-test-slave-pww0"
server_key: "47df4c8e517128ba36717c58b8a987a7"
server_key_iv: "28c72ddd11ed85f4b61e3313816176ef"
server_key_version: "encryptionkey@0"
I20260810 23:43:30.140792 30878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:43:30.145364 30916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:30.146947 30878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:43:30.147157 30878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "bb6b68019e1546aab67364314e7044ca"
format_stamp: "Formatted at 2026-08-10 23:43:30 on dist-test-slave-pww0"
server_key: "47df4c8e517128ba36717c58b8a987a7"
server_key_iv: "28c72ddd11ed85f4b61e3313816176ef"
server_key_version: "encryptionkey@0"
I20260810 23:43:30.147396 30878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:30.165429 30878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:30.170054 30878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:30.170368 30878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:30.171420 30878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:30.172919 30878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:30.173002 30878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:30.173130 30878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:30.173184 30878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:30.205560 30878 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43653
I20260810 23:43:30.205611 31136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43653 every 8 connection(s)
I20260810 23:43:30.206964 30878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:43:30.212719 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 30878
I20260810 23:43:30.212932 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:43:30.213598 30852 external_mini_cluster.cc:1428] Setting key 6df566a47b5b02901c5b56729283ad8d
Aug 10 23:43:30 dist-test-slave-pww0 krb5kdc[30677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405410, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:30.230726 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:30.209769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:50693 (local address 127.30.33.62:43615)
0810 23:43:30.210224 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:30.210230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:30.211263 (+ 1033us) server_negotiation.cc:407] Connection header received
0810 23:43:30.213324 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:30.213331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:30.213421 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:30.213521 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:30.215291 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.216163 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.218625 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.219084 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.219190 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:30.223750 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:30.223758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:30.223763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:30.223814 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:30.227176 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:30.228157 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:30.228163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:30.228166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:30.228334 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:30.228796 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:30.228801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:30.228804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:30.229151 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:30.229303 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:30.230091 (+ 788us) server_negotiation.cc:299] Negotiation successful
0810 23:43:30.230368 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260810 23:43:30.232039 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:30.210296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43615 (local address 127.30.33.1:50693)
0810 23:43:30.211086 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:30.211129 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:30.212984 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:30.213846 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:30.213856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:30.214465 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:30.215096 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:30.215118 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.216390 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.216396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:30.218429 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:30.218449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.219345 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.219352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:30.219708 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:30.220539 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:30.220570 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:30.223378 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:30.227397 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:30.227407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:30.227424 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:30.228008 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:30.228442 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:30.228445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:30.228448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:30.228675 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:30.229280 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:30.229288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:30.229677 (+ 389us) client_negotiation.cc:769] Sending connection context
0810 23:43:30.229927 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 23:43:30.230481 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":293,"threads_started":1}
I20260810 23:43:30.233922 31137 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43615
I20260810 23:43:30.234267 31137 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:30.235064 31137 heartbeater.cc:511] Master 127.30.33.62:43615 requested a full tablet report, sending...
I20260810 23:43:30.237430 30759 ts_manager.cc:194] Registered new tserver with Master: bb6b68019e1546aab67364314e7044ca (127.30.33.1:43653)
I20260810 23:43:30.239007 30759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:50693
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:43:30.288834 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43615
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:30.647861 31148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:30.648188 31148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:30.648319 31148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:30.656154 31148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:30.656309 31148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:30.656469 31148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:43:30.666850 31148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:30.668322 31148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:30.669772 31148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:30.684007 31169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:30.684017 31172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:30.684064 31168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:30.685107 31148 server_base.cc:1034] running on GCE node
I20260810 23:43:30.685766 31148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:30.686758 31148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:30.687978 31148 hybrid_clock.cc:648] HybridClock initialized: now 1786405410687920 us; error 46 us; skew 500 ppm
I20260810 23:43:30.688359 31148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:30 dist-test-slave-pww0 krb5kdc[30677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405410, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:30.691769 31148 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:43:30.693464 31148 webserver.cc:460] Webserver started at http://127.30.33.2:35001/ using document root <none> and password file <none>
I20260810 23:43:30.694214 31148 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:30.694343 31148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:30.694693 31148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:30.698024 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "7b32324fd7e64b5492e654c2f9e8748e"
format_stamp: "Formatted at 2026-08-10 23:43:30 on dist-test-slave-pww0"
server_key: "ccd14722c04b90218051a95abdcc0d8c"
server_key_iv: "a36d5c97b7d8e1e9320d8a1de2828f52"
server_key_version: "encryptionkey@0"
I20260810 23:43:30.698799 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "7b32324fd7e64b5492e654c2f9e8748e"
format_stamp: "Formatted at 2026-08-10 23:43:30 on dist-test-slave-pww0"
server_key: "ccd14722c04b90218051a95abdcc0d8c"
server_key_iv: "a36d5c97b7d8e1e9320d8a1de2828f52"
server_key_version: "encryptionkey@0"
I20260810 23:43:30.704568 31148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260810 23:43:30.708472 31188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:30.710021 31148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:30.710206 31148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "7b32324fd7e64b5492e654c2f9e8748e"
format_stamp: "Formatted at 2026-08-10 23:43:30 on dist-test-slave-pww0"
server_key: "ccd14722c04b90218051a95abdcc0d8c"
server_key_iv: "a36d5c97b7d8e1e9320d8a1de2828f52"
server_key_version: "encryptionkey@0"
I20260810 23:43:30.710443 31148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:30.735690 31148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:30.740137 31148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:30.740407 31148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:30.741420 31148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:30.742854 31148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:30.742934 31148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:30.743029 31148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:30.743072 31148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:30.775496 31148 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:37155
I20260810 23:43:30.775568 31396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:37155 every 8 connection(s)
I20260810 23:43:30.776860 31148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:43:30.782727 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31148
I20260810 23:43:30.782932 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:43:30.783357 30852 external_mini_cluster.cc:1428] Setting key e6fb6d08ea61ba0baa7b837097e627a6
Aug 10 23:43:30 dist-test-slave-pww0 krb5kdc[30677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405410, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:30.797240 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:30.779447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:42157 (local address 127.30.33.62:43615)
0810 23:43:30.779933 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:30.779938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:30.780974 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 23:43:30.782892 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:30.782898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:30.782986 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:30.783097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:30.784780 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.785575 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.787149 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.787629 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.787756 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:30.791364 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:30.791370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:30.791373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:30.791408 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:30.794009 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:30.794787 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:30.794792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:30.794794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:30.794927 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:30.795371 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:30.795375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:30.795377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:30.795696 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:30.795849 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:30.796580 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 23:43:30.796832 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":229,"threads_started":1}
I20260810 23:43:30.798252 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:30.780047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43615 (local address 127.30.33.2:42157)
0810 23:43:30.780800 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:30.780847 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:30.782589 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:30.783366 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:30.783376 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:30.783980 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:30.784590 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:30.784609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.785728 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.785733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:30.786995 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:30.787008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:30.787857 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:30.787864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:30.788104 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:30.788768 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:30.788792 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:30.791116 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:30.794160 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:30.794170 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:30.794190 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:30.794656 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:30.795038 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:30.795043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:30.795047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:30.795257 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:30.795840 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:30.795848 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:30.796202 (+ 354us) client_negotiation.cc:769] Sending connection context
0810 23:43:30.796472 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 23:43:30.797058 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":285,"threads_started":1}
I20260810 23:43:30.799826 31397 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43615
I20260810 23:43:30.800138 31397 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:30.800876 31397 heartbeater.cc:511] Master 127.30.33.62:43615 requested a full tablet report, sending...
I20260810 23:43:30.802481 30759 ts_manager.cc:194] Registered new tserver with Master: 7b32324fd7e64b5492e654c2f9e8748e (127.30.33.2:37155)
I20260810 23:43:30.803609 30759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:42157
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:43:30.852109 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43615
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:31.215063 31410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:31.215369 31410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:31.215547 31410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:31.223088 31410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:31.223222 31410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:31.223373 31410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:43:31.233652 31410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:43615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:31.235033 31410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:31.236393 31410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:43:31.242456 31137 heartbeater.cc:503] Master 127.30.33.62:43615 was elected leader, sending a full tablet report...
W20260810 23:43:31.250626 31417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:31.250665 31416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:31.250733 31420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:31.251437 31410 server_base.cc:1034] running on GCE node
I20260810 23:43:31.252454 31410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:31.253564 31410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:31.254783 31410 hybrid_clock.cc:648] HybridClock initialized: now 1786405411254729 us; error 51 us; skew 500 ppm
I20260810 23:43:31.255123 31410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:31 dist-test-slave-pww0 krb5kdc[30677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405411, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:31.258633 31410 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:43:31.260314 31410 webserver.cc:460] Webserver started at http://127.30.33.3:42899/ using document root <none> and password file <none>
I20260810 23:43:31.261101 31410 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:31.261253 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:31.261605 31410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:31.264822 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "f55fbec6580044cabaaf855ed1676217"
format_stamp: "Formatted at 2026-08-10 23:43:31 on dist-test-slave-pww0"
server_key: "30a52e203436b04ad535e226f6bf5aed"
server_key_iv: "c53aaaa73212f321b405bcaeae52a515"
server_key_version: "encryptionkey@0"
I20260810 23:43:31.265682 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "f55fbec6580044cabaaf855ed1676217"
format_stamp: "Formatted at 2026-08-10 23:43:31 on dist-test-slave-pww0"
server_key: "30a52e203436b04ad535e226f6bf5aed"
server_key_iv: "c53aaaa73212f321b405bcaeae52a515"
server_key_version: "encryptionkey@0"
I20260810 23:43:31.271858 31410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 23:43:31.275852 31431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:31.277376 31410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:31.277556 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "f55fbec6580044cabaaf855ed1676217"
format_stamp: "Formatted at 2026-08-10 23:43:31 on dist-test-slave-pww0"
server_key: "30a52e203436b04ad535e226f6bf5aed"
server_key_iv: "c53aaaa73212f321b405bcaeae52a515"
server_key_version: "encryptionkey@0"
I20260810 23:43:31.277776 31410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:31.292047 31410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:31.296766 31410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:31.297065 31410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:31.298126 31410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:31.299742 31410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:31.299829 31410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:31.299948 31410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:31.300002 31410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:31.330740 31410 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:34025
I20260810 23:43:31.330765 31645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:34025 every 8 connection(s)
I20260810 23:43:31.332140 31410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:43:31.336467 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31410
I20260810 23:43:31.336658 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:43:31.337110 30852 external_mini_cluster.cc:1428] Setting key 1a8f040a1e1c9a60ff1fc80cdc9570c7
Aug 10 23:43:31 dist-test-slave-pww0 krb5kdc[30677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405411, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:31.353353 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.334721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:57959 (local address 127.30.33.62:43615)
0810 23:43:31.335191 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.335196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.337625 (+ 2429us) server_negotiation.cc:407] Connection header received
0810 23:43:31.338185 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.338190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.338300 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.338415 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:31.340369 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.341171 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.342778 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.343185 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.343298 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.346938 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:31.346944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:31.346947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:31.346985 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:31.349706 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.350638 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.350643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.350645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.350788 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.351264 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.351269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.351272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.351676 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:31.351828 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.352679 (+ 851us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.352926 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260810 23:43:31.354382 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.335213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:43615 (local address 127.30.33.3:57959)
0810 23:43:31.336008 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:31.336050 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:31.337933 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:31.338705 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:31.338714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:31.339326 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:31.340196 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.340219 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.341327 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.341331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.342626 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.342641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.343416 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.343423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.343682 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.344337 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:31.344363 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:31.346675 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:31.349910 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:31.349919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:31.349934 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:31.350487 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:31.350906 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:31.350910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:31.350912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:31.351113 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:31.351828 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:31.351837 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:31.352259 (+ 422us) client_negotiation.cc:769] Sending connection context
0810 23:43:31.352540 (+ 281us) client_negotiation.cc:240] Negotiation successful
0810 23:43:31.353133 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":316,"threads_started":1}
I20260810 23:43:31.355855 31646 heartbeater.cc:347] Connected to a master server at 127.30.33.62:43615
I20260810 23:43:31.356187 31646 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:31.356997 31646 heartbeater.cc:511] Master 127.30.33.62:43615 requested a full tablet report, sending...
I20260810 23:43:31.358505 30759 ts_manager.cc:194] Registered new tserver with Master: f55fbec6580044cabaaf855ed1676217 (127.30.33.3:34025)
I20260810 23:43:31.359349 30759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:57959
I20260810 23:43:31.365294 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:43:31.383891 31651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.370113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.30.33.62:43615)
0810 23:43:31.370358 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.370365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.370713 (+ 348us) server_negotiation.cc:407] Connection header received
0810 23:43:31.370921 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.370925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.370992 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.371079 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:31.372646 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.373408 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.374869 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.375300 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.375418 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.378105 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:31.378110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:31.378113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:31.378152 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:31.380644 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.381492 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.381499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.381501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.381640 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.382123 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.382127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.382129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.382595 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:31.382840 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.383299 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.383570 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:43:31.388710 30759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:43:31.395880 30759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:43:31.434317 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.409316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:34025 (local address 127.0.0.1:34574)
0810 23:43:31.410305 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:31.410353 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:31.410543 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:31.425127 (+ 14584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:31.425137 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:31.425182 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:31.425658 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.425676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.428064 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.428072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.430782 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.430812 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.432967 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.432974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.433278 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.433379 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:43:31.433556 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 23:43:31.433837 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":3,"thread_start_us":523,"threads_started":1}
I20260810 23:43:31.434350 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.409636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.30.33.3:34025)
0810 23:43:31.424394 (+ 14758us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.424402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.424441 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:43:31.424548 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.424555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.424791 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.425044 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:31.425972 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.427857 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.431032 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.432679 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.432820 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.433082 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.433819 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.434007 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14564,"thread_start_us":14388,"threads_started":1}
I20260810 23:43:31.439327 31535 tablet_service.cc:1467] Processing CreateTablet for tablet abdd8a7df92546a9999ea10ff38a6002 (DEFAULT_TABLE table=test-table [id=ab463a0153ee43c48187a78f97c04655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:31.440897 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.410022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:37155 (local address 127.0.0.1:55816)
0810 23:43:31.432016 (+ 21994us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:31.432040 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:31.432196 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:31.433193 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:31.433198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:31.433247 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:31.434380 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.434396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.436071 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.436076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.438256 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.438275 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.439882 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.439889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.440052 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.440086 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:43:31.440295 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 23:43:31.440442 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21835,"mutex_wait_us":682,"thread_start_us":23470,"threads_started":1}
I20260810 23:43:31.441216 31535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd8a7df92546a9999ea10ff38a6002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:31.441864 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.410864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43653 (local address 127.0.0.1:33742)
0810 23:43:31.433802 (+ 22938us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:31.433822 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:31.433943 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:31.434723 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:31.434727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:31.434756 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:31.435137 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.435150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.437044 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.437049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.438872 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.438888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.441044 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.441050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.441253 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.441290 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:43:31.441381 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:43:31.441516 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22808,"mutex_wait_us":1,"thread_start_us":20835,"threads_started":1}
I20260810 23:43:31.442066 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.411210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.30.33.1:43653)
0810 23:43:31.426612 (+ 15402us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.426623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.434092 (+ 7469us) server_negotiation.cc:407] Connection header received
0810 23:43:31.434209 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.434217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.434517 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.434719 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:31.435356 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.436867 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.439063 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.440521 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.440657 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.440888 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.441591 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.441760 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15187,"thread_start_us":15410,"threads_started":1}
I20260810 23:43:31.442379 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.410591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.30.33.2:37155)
0810 23:43:31.432412 (+ 21821us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.432422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.432456 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:43:31.432565 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.432574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.432905 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.433141 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:31.434591 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.435940 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.438460 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.439676 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.439808 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.440029 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.441441 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.441634 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21637,"thread_start_us":21502,"threads_started":1}
I20260810 23:43:31.446074 31009 tablet_service.cc:1467] Processing CreateTablet for tablet abdd8a7df92546a9999ea10ff38a6002 (DEFAULT_TABLE table=test-table [id=ab463a0153ee43c48187a78f97c04655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:31.446074 31280 tablet_service.cc:1467] Processing CreateTablet for tablet abdd8a7df92546a9999ea10ff38a6002 (DEFAULT_TABLE table=test-table [id=ab463a0153ee43c48187a78f97c04655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:31.447679 31009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd8a7df92546a9999ea10ff38a6002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:31.447676 31280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd8a7df92546a9999ea10ff38a6002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:31.454620 31681 tablet_bootstrap.cc:492] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Bootstrap starting.
I20260810 23:43:31.459682 31681 tablet_bootstrap.cc:654] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:31.460965 31681 log.cc:824] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:31.464190 31685 tablet_bootstrap.cc:492] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Bootstrap starting.
I20260810 23:43:31.464334 31681 tablet_bootstrap.cc:492] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: No bootstrap required, opened a new log
I20260810 23:43:31.464555 31684 tablet_bootstrap.cc:492] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Bootstrap starting.
I20260810 23:43:31.464891 31681 ts_tablet_manager.cc:1397] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260810 23:43:31.468072 31684 tablet_bootstrap.cc:654] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:31.469164 31684 log.cc:824] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:31.469173 31685 tablet_bootstrap.cc:654] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:31.470489 31685 log.cc:824] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:31.472334 31684 tablet_bootstrap.cc:492] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: No bootstrap required, opened a new log
I20260810 23:43:31.472601 31684 ts_tablet_manager.cc:1397] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260810 23:43:31.473757 31685 tablet_bootstrap.cc:492] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: No bootstrap required, opened a new log
I20260810 23:43:31.474057 31685 ts_tablet_manager.cc:1397] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 23:43:31.475430 31681 raft_consensus.cc:348] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.475975 31681 raft_consensus.cc:374] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:31.476082 31681 raft_consensus.cc:729] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55fbec6580044cabaaf855ed1676217, State: Initialized, Role: FOLLOWER
I20260810 23:43:31.476758 31681 consensus_queue.cc:260] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.478003 31646 heartbeater.cc:503] Master 127.30.33.62:43615 was elected leader, sending a full tablet report...
I20260810 23:43:31.480221 31681 ts_tablet_manager.cc:1428] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260810 23:43:31.481120 31684 raft_consensus.cc:348] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.481518 31684 raft_consensus.cc:374] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:31.481642 31684 raft_consensus.cc:729] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b32324fd7e64b5492e654c2f9e8748e, State: Initialized, Role: FOLLOWER
I20260810 23:43:31.482277 31684 consensus_queue.cc:260] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.484653 31397 heartbeater.cc:503] Master 127.30.33.62:43615 was elected leader, sending a full tablet report...
I20260810 23:43:31.485505 31684 ts_tablet_manager.cc:1428] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260810 23:43:31.485754 31685 raft_consensus.cc:348] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.486255 31685 raft_consensus.cc:374] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:31.486377 31685 raft_consensus.cc:729] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6b68019e1546aab67364314e7044ca, State: Initialized, Role: FOLLOWER
I20260810 23:43:31.487069 31685 consensus_queue.cc:260] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.488857 31685 ts_tablet_manager.cc:1428] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260810 23:43:31.528932 31398 tablet.cc:2369] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:31.544378 31694 raft_consensus.cc:482] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:43:31.544697 31694 raft_consensus.cc:504] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.546497 31694 leader_election.cc:288] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb6b68019e1546aab67364314e7044ca (127.30.33.1:43653), 7b32324fd7e64b5492e654c2f9e8748e (127.30.33.2:37155)
I20260810 23:43:31.553033 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.546937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:42409 (local address 127.30.33.1:43653)
0810 23:43:31.547199 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.547206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.547234 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:43:31.547438 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.547444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.547555 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.547635 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:31.548140 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.548920 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.550842 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.552164 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.552292 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.552400 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.552662 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.552790 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 23:43:31.553081 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.546742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43653 (local address 127.30.33.3:42409)
0810 23:43:31.547044 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:31.547066 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:31.547206 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:31.547697 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:31.547702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:31.547740 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:31.547943 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.547956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.549059 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.549064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.550666 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.550680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.552345 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.552351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.552512 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.552536 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:43:31.552621 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:43:31.552761 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260810 23:43:31.553933 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.546843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:37155 (local address 127.30.33.3:39991)
0810 23:43:31.547763 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:31.547778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:31.547944 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:31.548263 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:31.548267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:31.548292 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:31.548580 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.548588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.549930 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.549934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.551766 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:31.551782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.553252 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.553261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:31.553406 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.553440 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:43:31.553535 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:43:31.553662 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":536,"threads_started":1}
I20260810 23:43:31.553942 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.547234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:39991 (local address 127.30.33.2:37155)
0810 23:43:31.547438 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.547444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.547899 (+ 455us) server_negotiation.cc:407] Connection header received
0810 23:43:31.548079 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.548084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.548148 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.548234 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:31.548748 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.549761 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.551920 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.552890 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.553024 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.553118 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.553594 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.553694 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:43:31.554320 31051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd8a7df92546a9999ea10ff38a6002" candidate_uuid: "f55fbec6580044cabaaf855ed1676217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6b68019e1546aab67364314e7044ca" is_pre_election: true
I20260810 23:43:31.554850 31051 raft_consensus.cc:2454] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55fbec6580044cabaaf855ed1676217 in term 0.
I20260810 23:43:31.554939 31314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd8a7df92546a9999ea10ff38a6002" candidate_uuid: "f55fbec6580044cabaaf855ed1676217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b32324fd7e64b5492e654c2f9e8748e" is_pre_election: true
I20260810 23:43:31.555508 31314 raft_consensus.cc:2454] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55fbec6580044cabaaf855ed1676217 in term 0.
I20260810 23:43:31.555562 31435 leader_election.cc:302] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb6b68019e1546aab67364314e7044ca, f55fbec6580044cabaaf855ed1676217; no voters:
I20260810 23:43:31.555965 31694 raft_consensus.cc:2781] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:43:31.556103 31694 raft_consensus.cc:482] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:43:31.556212 31694 raft_consensus.cc:3037] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:31.558485 31694 raft_consensus.cc:504] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.559091 31694 leader_election.cc:288] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [CANDIDATE]: Term 1 election: Requested vote from peers bb6b68019e1546aab67364314e7044ca (127.30.33.1:43653), 7b32324fd7e64b5492e654c2f9e8748e (127.30.33.2:37155)
I20260810 23:43:31.559652 31314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd8a7df92546a9999ea10ff38a6002" candidate_uuid: "f55fbec6580044cabaaf855ed1676217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b32324fd7e64b5492e654c2f9e8748e"
I20260810 23:43:31.559697 31051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd8a7df92546a9999ea10ff38a6002" candidate_uuid: "f55fbec6580044cabaaf855ed1676217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6b68019e1546aab67364314e7044ca"
I20260810 23:43:31.559871 31314 raft_consensus.cc:3037] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:31.559969 31051 raft_consensus.cc:3037] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:31.562242 31314 raft_consensus.cc:2454] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55fbec6580044cabaaf855ed1676217 in term 1.
I20260810 23:43:31.562430 31051 raft_consensus.cc:2454] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55fbec6580044cabaaf855ed1676217 in term 1.
I20260810 23:43:31.562675 31433 leader_election.cc:302] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b32324fd7e64b5492e654c2f9e8748e, f55fbec6580044cabaaf855ed1676217; no voters:
I20260810 23:43:31.563010 31694 raft_consensus.cc:2781] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:31.563485 31694 raft_consensus.cc:686] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 1 LEADER]: Becoming Leader. State: Replica: f55fbec6580044cabaaf855ed1676217, State: Running, Role: LEADER
I20260810 23:43:31.564043 31694 consensus_queue.cc:237] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } }
I20260810 23:43:31.568866 30759 catalog_manager.cc:5495] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 reported cstate change: term changed from 0 to 1, leader changed from <none> to f55fbec6580044cabaaf855ed1676217 (127.30.33.3). New cstate: current_term: 1 leader_uuid: "f55fbec6580044cabaaf855ed1676217" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55fbec6580044cabaaf855ed1676217" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 34025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 } health_report { overall_health: UNKNOWN } } }
W20260810 23:43:31.583971 31649 tablet.cc:2369] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:31.602785 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.597472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.30.33.3:34025)
0810 23:43:31.597672 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.597677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.597694 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:43:31.597785 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.597789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.597865 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.597993 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:43:31.598339 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.599055 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.600633 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.601042 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.601142 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.601723 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:43:31.602266 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:43:31.602380 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.602454 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.602529 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:43:31.611124 31051 raft_consensus.cc:1261] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 1 FOLLOWER]: Refusing update from remote peer f55fbec6580044cabaaf855ed1676217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:31.611124 31314 raft_consensus.cc:1261] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 1 FOLLOWER]: Refusing update from remote peer f55fbec6580044cabaaf855ed1676217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:31.612112 31698 consensus_queue.cc:1035] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6b68019e1546aab67364314e7044ca" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:31.612493 31694 consensus_queue.cc:1035] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b32324fd7e64b5492e654c2f9e8748e" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:31.639219 30755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:31.639559 30755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:31.643486 30755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79650e3be93f41e69e6d3ddd57564d0a: Sending DeleteTablet for 3 replicas of tablet abdd8a7df92546a9999ea10ff38a6002
I20260810 23:43:31.644835 31535 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd8a7df92546a9999ea10ff38a6002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:31 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:34574
I20260810 23:43:31.645296 31280 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd8a7df92546a9999ea10ff38a6002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:31 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:55816
I20260810 23:43:31.646181 31009 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd8a7df92546a9999ea10ff38a6002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:31 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:33742
I20260810 23:43:31.647899 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 30878
I20260810 23:43:31.656102 31731 tablet_replica.cc:330] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: stopping tablet replica
I20260810 23:43:31.656490 31730 tablet_replica.cc:330] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: stopping tablet replica
I20260810 23:43:31.656833 31731 raft_consensus.cc:2229] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:31.657137 31730 raft_consensus.cc:2229] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:31.657517 31731 raft_consensus.cc:2258] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:31.657838 31730 raft_consensus.cc:2258] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:31.660179 31731 ts_tablet_manager.cc:1905] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:31.660249 31730 ts_tablet_manager.cc:1905] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:31.667840 31730 ts_tablet_manager.cc:1918] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:31.667838 31731 ts_tablet_manager.cc:1918] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:31.668056 31731 log.cc:1196] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/abdd8a7df92546a9999ea10ff38a6002
I20260810 23:43:31.668252 31730 log.cc:1196] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/abdd8a7df92546a9999ea10ff38a6002
I20260810 23:43:31.668737 31731 ts_tablet_manager.cc:1939] T abdd8a7df92546a9999ea10ff38a6002 P bb6b68019e1546aab67364314e7044ca: Deleting consensus metadata
I20260810 23:43:31.668982 31730 ts_tablet_manager.cc:1939] T abdd8a7df92546a9999ea10ff38a6002 P 7b32324fd7e64b5492e654c2f9e8748e: Deleting consensus metadata
I20260810 23:43:31.671015 30721 catalog_manager.cc:4841] TS 7b32324fd7e64b5492e654c2f9e8748e (127.30.33.2:37155): tablet abdd8a7df92546a9999ea10ff38a6002 (table test-table [id=ab463a0153ee43c48187a78f97c04655]) successfully deleted
I20260810 23:43:31.670981 30728 catalog_manager.cc:4841] TS bb6b68019e1546aab67364314e7044ca (127.30.33.1:43653): tablet abdd8a7df92546a9999ea10ff38a6002 (table test-table [id=ab463a0153ee43c48187a78f97c04655]) successfully deleted
I20260810 23:43:31.672180 31727 tablet_replica.cc:330] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: stopping tablet replica
I20260810 23:43:31.672701 31727 raft_consensus.cc:2229] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:43:31.673466 31727 raft_consensus.cc:2258] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:31.675886 31727 ts_tablet_manager.cc:1905] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:43:31 dist-test-slave-pww0 krb5kdc[30677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:43:31.683005 31727 ts_tablet_manager.cc:1918] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:31.683190 31727 log.cc:1196] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/abdd8a7df92546a9999ea10ff38a6002
I20260810 23:43:31.683750 31727 ts_tablet_manager.cc:1939] T abdd8a7df92546a9999ea10ff38a6002 P f55fbec6580044cabaaf855ed1676217: Deleting consensus metadata
I20260810 23:43:31.685509 30728 catalog_manager.cc:4841] TS f55fbec6580044cabaaf855ed1676217 (127.30.33.3:34025): tablet abdd8a7df92546a9999ea10ff38a6002 (table test-table [id=ab463a0153ee43c48187a78f97c04655]) successfully deleted
I20260810 23:43:31.687954 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.649482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.30.33.1:43653)
0810 23:43:31.649699 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.649706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.667574 (+ 17868us) server_negotiation.cc:407] Connection header received
0810 23:43:31.668204 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.668211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.668315 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.668406 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:31.670443 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.671500 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.673372 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.674033 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.674185 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.680079 (+ 5894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:31.680105 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:31.680127 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:31.680189 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:31.684514 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.685606 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.685617 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.685620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.685816 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.686239 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.686244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.686247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.686756 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:31.687082 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.687287 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.687563 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:43:31.688778 31123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:31.783527 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 30878
I20260810 23:43:31.816736 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31148
Aug 10 23:43:31 dist-test-slave-pww0 krb5kdc[30677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:43:31.834555 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.817575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.30.33.2:37155)
0810 23:43:31.817755 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.817760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.817834 (+ 74us) server_negotiation.cc:407] Connection header received
0810 23:43:31.818025 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.818029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.818114 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.818215 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:31.819513 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.820273 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.821631 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.822041 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.822167 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.827019 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:31.827043 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:31.827064 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:31.827118 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:31.831022 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.832100 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.832112 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.832116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.832305 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.832675 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.832680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.832684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.833237 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:31.833565 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.833827 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.834082 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:43:31.835362 31386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:31.927714 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31148
I20260810 23:43:31.959561 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31410
Aug 10 23:43:31 dist-test-slave-pww0 krb5kdc[30677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:43:31.980839 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:31.960395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.30.33.3:34025)
0810 23:43:31.960587 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:31.960591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:31.960612 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:43:31.960838 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:31.960843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:31.960915 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:31.961014 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:31.962946 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.963896 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.965912 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:31.966422 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:31.966541 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:31.972661 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:31.972686 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:31.972708 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:31.972775 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:31.977090 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.978071 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.978084 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.978088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.978335 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:31.978762 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:31.978768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:31.978772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:31.979362 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:31.979774 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:31.980058 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 23:43:31.980326 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:43:31.981689 31639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:32.110172 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31410
I20260810 23:43:32.140089 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 30701
I20260810 23:43:32.141283 30821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:32.255074 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 30701
2026-08-10T23:43:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4893 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T23:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:32Z Disabled control of system clock
I20260810 23:43:32.357914 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40289
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:40289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:43:32.716506 31746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:32.716830 31746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:32.716913 31746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:32.724414 31746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:32.724551 31746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:32.724591 31746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:32.724642 31746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:32.724685 31746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:32.734382 31746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:40289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:32.735788 31746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:32.737602 31746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:32.749688 31757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:32.749637 31756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:32.749728 31759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:32.750568 31746 server_base.cc:1034] running on GCE node
I20260810 23:43:32.751448 31746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:32.753046 31746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:32.754271 31746 hybrid_clock.cc:648] HybridClock initialized: now 1786405412754206 us; error 58 us; skew 500 ppm
I20260810 23:43:32.754659 31746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:32.762579 31746 webserver.cc:460] Webserver started at http://127.30.33.62:46327/ using document root <none> and password file <none>
I20260810 23:43:32.763409 31746 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:32.763571 31746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:32.763939 31746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:32.767138 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4"
format_stamp: "Formatted at 2026-08-10 23:43:32 on dist-test-slave-pww0"
server_key: "3ba4b7061c65bfb0888e8a4e18c7ffe8"
server_key_iv: "188dfa7791ea1c1dac8e89b198135595"
server_key_version: "encryptionkey@0"
I20260810 23:43:32.768126 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4"
format_stamp: "Formatted at 2026-08-10 23:43:32 on dist-test-slave-pww0"
server_key: "3ba4b7061c65bfb0888e8a4e18c7ffe8"
server_key_iv: "188dfa7791ea1c1dac8e89b198135595"
server_key_version: "encryptionkey@0"
I20260810 23:43:32.773984 31746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:43:32.778216 31775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:32.779763 31746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:32.779959 31746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4"
format_stamp: "Formatted at 2026-08-10 23:43:32 on dist-test-slave-pww0"
server_key: "3ba4b7061c65bfb0888e8a4e18c7ffe8"
server_key_iv: "188dfa7791ea1c1dac8e89b198135595"
server_key_version: "encryptionkey@0"
I20260810 23:43:32.780179 31746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:32.809911 31746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:32.810868 31746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:32.811162 31746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:32.829766 31746 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:40289
I20260810 23:43:32.829787 31862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:40289 every 8 connection(s)
I20260810 23:43:32.831218 31746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:32.832456 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31746
I20260810 23:43:32.832624 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:32.833108 30852 external_mini_cluster.cc:1428] Setting key 118e9d2c364f959aa2a4a06432edd5c2
I20260810 23:43:32.835960 31863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:32.850024 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:32.835233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.30.33.62:40289)
0810 23:43:32.836447 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:32.836463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:32.836520 (+ 57us) server_negotiation.cc:407] Connection header received
0810 23:43:32.838851 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:32.838883 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:32.839608 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:32.840112 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:32.840716 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:32.842812 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:32.844361 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:32.845474 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:32.845679 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:32.846819 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:32.846830 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:32.846852 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:32.847700 (+ 848us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:32.847834 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:32.847863 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:32.848007 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:32.848219 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 23:43:32.848622 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":422,"threads_started":1}
I20260810 23:43:32.850919 31863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4: Bootstrap starting.
I20260810 23:43:32.854853 31863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:32.856417 31863 log.cc:824] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:32.859368 31863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4: No bootstrap required, opened a new log
I20260810 23:43:32.867184 31863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40289 } }
I20260810 23:43:32.867640 31863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:32.867726 31863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfccf07bba74e7c8bfc64f9f8af7df4, State: Initialized, Role: FOLLOWER
I20260810 23:43:32.868311 31863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40289 } }
I20260810 23:43:32.868528 31863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:32.868660 31863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:32.868806 31863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:32.871562 31863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40289 } }
I20260810 23:43:32.872133 31863 leader_election.cc:302] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cfccf07bba74e7c8bfc64f9f8af7df4; no voters:
I20260810 23:43:32.872699 31863 leader_election.cc:288] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:32.872840 31872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:32.873581 31872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [term 1 LEADER]: Becoming Leader. State: Replica: 7cfccf07bba74e7c8bfc64f9f8af7df4, State: Running, Role: LEADER
I20260810 23:43:32.874112 31872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40289 } }
I20260810 23:43:32.874766 31863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:32.876967 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40289 } } }
I20260810 23:43:32.877434 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:32.877763 31875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cfccf07bba74e7c8bfc64f9f8af7df4. Latest consensus state: current_term: 1 leader_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfccf07bba74e7c8bfc64f9f8af7df4" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40289 } } }
I20260810 23:43:32.878109 31875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:32.879894 31886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:32.885253 31886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:32.895087 31886 catalog_manager.cc:1347] Generated new cluster ID: 7eafe4e425364130b7eea0f2e02ac470
I20260810 23:43:32.895210 31886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:32.909642 31886 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:32.911042 31886 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:32.917960 31886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4: Generated new TSK 0
I20260810 23:43:32.918855 31886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:43:32.928912 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40289
--builtin_ntp_servers=127.30.33.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:43:33.343053 31910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:33.343367 31910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:33.343501 31910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:33.351197 31910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:33.351312 31910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:33.351477 31910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:43:33.361142 31910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:33.362546 31910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:33.364512 31910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:33.378674 31926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:33.378608 31923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:33.378628 31920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:33.379643 31910 server_base.cc:1034] running on GCE node
I20260810 23:43:33.380358 31910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:33.381500 31910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:33.382740 31910 hybrid_clock.cc:648] HybridClock initialized: now 1786405413382690 us; error 36 us; skew 500 ppm
I20260810 23:43:33.383105 31910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:33.385773 31910 webserver.cc:460] Webserver started at http://127.30.33.1:35311/ using document root <none> and password file <none>
I20260810 23:43:33.386549 31910 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:33.386652 31910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:33.386973 31910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:33.390173 31910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "4fa8430ea8434bf4b9b01793b09da8f8"
format_stamp: "Formatted at 2026-08-10 23:43:33 on dist-test-slave-pww0"
server_key: "fab195d74a7976ebb0d3ea47b3363796"
server_key_iv: "f931585204195d89d0307d10082a7db9"
server_key_version: "encryptionkey@0"
I20260810 23:43:33.391091 31910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "4fa8430ea8434bf4b9b01793b09da8f8"
format_stamp: "Formatted at 2026-08-10 23:43:33 on dist-test-slave-pww0"
server_key: "fab195d74a7976ebb0d3ea47b3363796"
server_key_iv: "f931585204195d89d0307d10082a7db9"
server_key_version: "encryptionkey@0"
I20260810 23:43:33.396858 31910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 23:43:33.400914 31934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:33.402416 31910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:33.402606 31910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "4fa8430ea8434bf4b9b01793b09da8f8"
format_stamp: "Formatted at 2026-08-10 23:43:33 on dist-test-slave-pww0"
server_key: "fab195d74a7976ebb0d3ea47b3363796"
server_key_iv: "f931585204195d89d0307d10082a7db9"
server_key_version: "encryptionkey@0"
I20260810 23:43:33.402858 31910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:33.438710 31910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:33.439683 31910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:33.439958 31910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:33.441035 31910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:33.442646 31910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:33.442752 31910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:33.442862 31910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:33.442931 31910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:33.474884 31910 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:37857
I20260810 23:43:33.474992 32124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:37857 every 8 connection(s)
I20260810 23:43:33.476362 31910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:43:33.479141 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 31910
I20260810 23:43:33.479359 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:43:33.479907 30852 external_mini_cluster.cc:1428] Setting key d09bbffd60535cc19af9c06d991c1dbc
I20260810 23:43:33.483258 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40289
--builtin_ntp_servers=127.30.33.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:43:33.490108 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:33.478958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:57973 (local address 127.30.33.62:40289)
0810 23:43:33.479528 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:33.479535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:33.481037 (+ 1502us) server_negotiation.cc:407] Connection header received
0810 23:43:33.483385 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:33.483391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:33.483508 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:33.483618 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:33.484917 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.485676 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.487297 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.487747 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.487863 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:33.489264 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:33.489269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:33.489272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:33.489292 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:33.489333 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:33.489346 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:33.489422 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:33.489648 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 23:43:33.489763 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":258,"threads_started":1}
I20260810 23:43:33.491202 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:33.479712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40289 (local address 127.30.33.1:57973)
0810 23:43:33.480875 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:33.480922 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:33.483076 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:33.483898 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:33.483907 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:33.484049 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:33.484753 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:33.484774 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.485846 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.485849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:33.487139 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:33.487152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.487977 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.487984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:33.488226 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:33.488929 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:33.488932 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:33.488974 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:33.489068 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:33.489436 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:33.489449 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:33.489457 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:43:33.489623 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 23:43:33.490057 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260810 23:43:33.492836 32126 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40289
I20260810 23:43:33.493175 32126 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:33.494113 32126 heartbeater.cc:511] Master 127.30.33.62:40289 requested a full tablet report, sending...
I20260810 23:43:33.496757 31808 ts_manager.cc:194] Registered new tserver with Master: 4fa8430ea8434bf4b9b01793b09da8f8 (127.30.33.1:37857)
I20260810 23:43:33.498658 31808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.1:57973
W20260810 23:43:33.844954 32132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:33.845275 32132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:33.845368 32132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:33.852991 32132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:33.853111 32132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:33.853220 32132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:43:33.862818 32132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:33.864235 32132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:33.866039 32132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:33.879961 32147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:33.879900 32143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:33.880002 32141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:33.880872 32132 server_base.cc:1034] running on GCE node
I20260810 23:43:33.882037 32132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:33.883111 32132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:33.884347 32132 hybrid_clock.cc:648] HybridClock initialized: now 1786405413884276 us; error 58 us; skew 500 ppm
I20260810 23:43:33.884702 32132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:33.887298 32132 webserver.cc:460] Webserver started at http://127.30.33.2:43451/ using document root <none> and password file <none>
I20260810 23:43:33.888094 32132 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:33.888211 32132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:33.888548 32132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:33.891778 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad"
format_stamp: "Formatted at 2026-08-10 23:43:33 on dist-test-slave-pww0"
server_key: "595462d09682e078495f5a226cf9858c"
server_key_iv: "b2f23129f9ad425aff5b84742e05cd02"
server_key_version: "encryptionkey@0"
I20260810 23:43:33.892653 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad"
format_stamp: "Formatted at 2026-08-10 23:43:33 on dist-test-slave-pww0"
server_key: "595462d09682e078495f5a226cf9858c"
server_key_iv: "b2f23129f9ad425aff5b84742e05cd02"
server_key_version: "encryptionkey@0"
I20260810 23:43:33.898398 32132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:43:33.902456 32164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:33.904034 32132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:33.904225 32132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad"
format_stamp: "Formatted at 2026-08-10 23:43:33 on dist-test-slave-pww0"
server_key: "595462d09682e078495f5a226cf9858c"
server_key_iv: "b2f23129f9ad425aff5b84742e05cd02"
server_key_version: "encryptionkey@0"
I20260810 23:43:33.904462 32132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:33.915851 32132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:33.916738 32132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:33.917016 32132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:33.918187 32132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:33.919860 32132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:33.919967 32132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:33.920090 32132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:33.920177 32132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:33.952371 32132 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:44307
I20260810 23:43:33.952430 32349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:44307 every 8 connection(s)
I20260810 23:43:33.953855 32132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:43:33.958818 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 32132
I20260810 23:43:33.959033 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:43:33.959674 30852 external_mini_cluster.cc:1428] Setting key 737e48fabca8ca526375700846d3afa6
I20260810 23:43:33.963523 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40289
--builtin_ntp_servers=127.30.33.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:43:33.970541 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:33.957020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:60027 (local address 127.30.33.62:40289)
0810 23:43:33.957246 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:33.957251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:33.958992 (+ 1741us) server_negotiation.cc:407] Connection header received
0810 23:43:33.961612 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:33.961617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:33.961703 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:33.961809 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:33.963678 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.964512 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.967055 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.967558 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.967674 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:33.969602 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:33.969607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:33.969611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:33.969635 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:33.969673 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:33.969687 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:33.969785 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:33.970128 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 23:43:33.970277 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 23:43:33.972046 32355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:33.957543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40289 (local address 127.30.33.2:60027)
0810 23:43:33.958747 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:33.958806 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:33.961256 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:33.962170 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:33.962185 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:33.962353 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:33.963357 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:33.963395 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.964725 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.964733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:33.966853 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:33.966876 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:33.967839 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:33.967848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:33.968193 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:33.969110 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:33.969114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:33.969189 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:33.969316 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:33.969814 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:33.969827 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:33.969835 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:43:33.970082 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 23:43:33.970603 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":322,"threads_started":1}
I20260810 23:43:33.974073 32350 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40289
I20260810 23:43:33.974444 32350 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:33.975389 32350 heartbeater.cc:511] Master 127.30.33.62:40289 requested a full tablet report, sending...
I20260810 23:43:33.977344 31808 ts_manager.cc:194] Registered new tserver with Master: c4a0e7f90b234c9eaa5991aa56f0c3ad (127.30.33.2:44307)
I20260810 23:43:33.978461 31808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.2:60027
W20260810 23:43:34.324450 32362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:34.324786 32362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:34.324914 32362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:34.332465 32362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:34.332604 32362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:34.332741 32362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:43:34.342166 32362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:40289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:34.343588 32362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:34.345453 32362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:34.358989 32372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:34.359098 32368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:34.359045 32367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:34.359970 32362 server_base.cc:1034] running on GCE node
I20260810 23:43:34.360626 32362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:34.361647 32362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:34.362905 32362 hybrid_clock.cc:648] HybridClock initialized: now 1786405414362841 us; error 50 us; skew 500 ppm
I20260810 23:43:34.363268 32362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:34.365790 32362 webserver.cc:460] Webserver started at http://127.30.33.3:37229/ using document root <none> and password file <none>
I20260810 23:43:34.366528 32362 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:34.366648 32362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:34.366966 32362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:34.370132 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "daf1e1d22e454edb84684da211dea728"
format_stamp: "Formatted at 2026-08-10 23:43:34 on dist-test-slave-pww0"
server_key: "f22dd0d1a317ed8c2dbc915410f27f13"
server_key_iv: "420b098ad4b56fb10b406a4821c0f456"
server_key_version: "encryptionkey@0"
I20260810 23:43:34.371019 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "daf1e1d22e454edb84684da211dea728"
format_stamp: "Formatted at 2026-08-10 23:43:34 on dist-test-slave-pww0"
server_key: "f22dd0d1a317ed8c2dbc915410f27f13"
server_key_iv: "420b098ad4b56fb10b406a4821c0f456"
server_key_version: "encryptionkey@0"
I20260810 23:43:34.377027 32362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 23:43:34.381135 32383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:34.382710 32362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:34.382920 32362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "daf1e1d22e454edb84684da211dea728"
format_stamp: "Formatted at 2026-08-10 23:43:34 on dist-test-slave-pww0"
server_key: "f22dd0d1a317ed8c2dbc915410f27f13"
server_key_iv: "420b098ad4b56fb10b406a4821c0f456"
server_key_version: "encryptionkey@0"
I20260810 23:43:34.383150 32362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:34.405333 32362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:34.406304 32362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:34.406610 32362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:34.407773 32362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:34.409543 32362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:34.409648 32362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:34.409773 32362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:34.409848 32362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:34.442018 32362 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:40163
I20260810 23:43:34.442101 32603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:40163 every 8 connection(s)
I20260810 23:43:34.443428 32362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:43:34.450801 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 32362
I20260810 23:43:34.450980 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:43:34.451618 30852 external_mini_cluster.cc:1428] Setting key d807fafb893dc7a60796bb7e3ad85539
I20260810 23:43:34.457165 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.446179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:34985 (local address 127.30.33.62:40289)
0810 23:43:34.446429 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.446436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.448023 (+ 1587us) server_negotiation.cc:407] Connection header received
0810 23:43:34.449813 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.449818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.449893 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.449989 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:34.451175 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.452236 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.454383 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.454935 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.455036 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.456464 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:34.456468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:34.456472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:34.456492 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:34.456524 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:34.456537 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:34.456611 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.456804 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.456896 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:43:34.458247 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.446807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:40289 (local address 127.30.33.3:34985)
0810 23:43:34.447848 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.447897 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:34.449545 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:34.450197 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:34.450206 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:34.450335 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:34.450956 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.450981 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.452426 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.452430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.454223 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.454240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.455150 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.455156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.455384 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.456127 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:34.456130 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:34.456174 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:34.456273 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:34.456615 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:34.456621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:34.456627 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:43:34.456790 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 23:43:34.457127 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":294,"threads_started":1}
I20260810 23:43:34.459676 32607 heartbeater.cc:347] Connected to a master server at 127.30.33.62:40289
I20260810 23:43:34.459986 32607 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:34.460762 32607 heartbeater.cc:511] Master 127.30.33.62:40289 requested a full tablet report, sending...
I20260810 23:43:34.462230 31808 ts_manager.cc:194] Registered new tserver with Master: daf1e1d22e454edb84684da211dea728 (127.30.33.3:40163)
I20260810 23:43:34.463037 31808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.3:34985
I20260810 23:43:34.467527 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:43:34.478319 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.471880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.30.33.62:40289)
0810 23:43:34.472134 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.472140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.472537 (+ 397us) server_negotiation.cc:407] Connection header received
0810 23:43:34.472757 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.472761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.472831 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.472918 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:34.473475 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.474203 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.475914 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.476353 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.476472 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.477605 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:34.477610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:34.477613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:34.477635 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:34.477669 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:34.477683 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:34.477770 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.477907 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.478014 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:43:34.482386 31808 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:43:34.489692 31808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:43:34.502214 32126 heartbeater.cc:503] Master 127.30.33.62:40289 was elected leader, sending a full tablet report...
I20260810 23:43:34.515318 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.502768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:40163 (local address 127.0.0.1:58584)
0810 23:43:34.503642 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.503690 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:34.503920 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:34.506059 (+ 2139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:34.506067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:34.506106 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:34.506571 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.506588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.509057 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.509064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.511875 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.511895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.514166 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.514174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.514459 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.514530 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 23:43:34.514657 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:43:34.514905 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":1448,"threads_started":1}
I20260810 23:43:34.518748 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.503019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.30.33.3:40163)
0810 23:43:34.505359 (+ 2340us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.505369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.505409 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:43:34.505509 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.505517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.505798 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.506798 (+ 1000us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:34.507101 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.508868 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.512186 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.513887 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.517642 (+ 3755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.518015 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.518107 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.518337 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"thread_start_us":2352,"threads_started":1}
I20260810 23:43:34.518919 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.502804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:44307 (local address 127.0.0.1:50318)
0810 23:43:34.504511 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.504527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:34.504635 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:34.507320 (+ 2685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:34.507326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:34.507356 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:34.508763 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.508782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.510974 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.510980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.514811 (+ 3831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.514831 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.518010 (+ 3179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.518018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.518261 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.518300 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:43:34.518386 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:43:34.518542 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"thread_start_us":779,"threads_started":1}
I20260810 23:43:34.519968 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.503412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.30.33.2:44307)
0810 23:43:34.506258 (+ 2846us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.506267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.506298 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:43:34.506412 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.506418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.506773 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.506989 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:34.509023 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.510777 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.515108 (+ 4331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.517378 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.517527 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.517841 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.519328 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.519583 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2627,"thread_start_us":2501,"threads_started":1}
I20260810 23:43:34.523437 32478 tablet_service.cc:1467] Processing CreateTablet for tablet b585059013c3488eb152b2a4ff743313 (DEFAULT_TABLE table=test-table [id=d34a1ebff4d14905a47987d9bb6b8dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:34.524036 32242 tablet_service.cc:1467] Processing CreateTablet for tablet b585059013c3488eb152b2a4ff743313 (DEFAULT_TABLE table=test-table [id=d34a1ebff4d14905a47987d9bb6b8dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:34.525076 32478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b585059013c3488eb152b2a4ff743313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:34.525350 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b585059013c3488eb152b2a4ff743313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:34.527385 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.506455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43668)
0810 23:43:34.515673 (+ 9218us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.515696 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:34.515829 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:34.520478 (+ 4649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:34.520482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:34.520515 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:34.520699 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.520711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.522948 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.522954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.524741 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.524759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.526600 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.526607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.526807 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.526844 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:43:34.526941 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:43:34.527093 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9077,"thread_start_us":13841,"threads_started":1}
I20260810 23:43:34.527606 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.506802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.30.33.1:37857)
0810 23:43:34.519778 (+ 12976us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.519787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.519819 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:43:34.519923 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.519931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.520228 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.520916 (+ 688us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:34.521153 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.522725 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.524927 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.526404 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.526552 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.526840 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.527025 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.527196 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12774,"thread_start_us":13447,"threads_started":1}
I20260810 23:43:34.532759 32011 tablet_service.cc:1467] Processing CreateTablet for tablet b585059013c3488eb152b2a4ff743313 (DEFAULT_TABLE table=test-table [id=d34a1ebff4d14905a47987d9bb6b8dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:34.534583 32011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b585059013c3488eb152b2a4ff743313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:34.537815 32651 tablet_bootstrap.cc:492] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Bootstrap starting.
I20260810 23:43:34.541244 32651 tablet_bootstrap.cc:654] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:34.542510 32653 tablet_bootstrap.cc:492] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Bootstrap starting.
I20260810 23:43:34.542753 32651 log.cc:824] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:34.546540 32653 tablet_bootstrap.cc:654] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:34.546587 32651 tablet_bootstrap.cc:492] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: No bootstrap required, opened a new log
I20260810 23:43:34.546912 32651 ts_tablet_manager.cc:1397] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260810 23:43:34.548024 32653 log.cc:824] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:34.551153 32660 tablet_bootstrap.cc:492] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Bootstrap starting.
I20260810 23:43:34.551659 32653 tablet_bootstrap.cc:492] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: No bootstrap required, opened a new log
I20260810 23:43:34.552026 32653 ts_tablet_manager.cc:1397] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260810 23:43:34.554775 32660 tablet_bootstrap.cc:654] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:34.556313 32660 log.cc:824] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:34.557363 32651 raft_consensus.cc:348] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.557871 32651 raft_consensus.cc:374] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:34.558020 32651 raft_consensus.cc:729] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf1e1d22e454edb84684da211dea728, State: Initialized, Role: FOLLOWER
I20260810 23:43:34.558746 32651 consensus_queue.cc:260] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.560050 32660 tablet_bootstrap.cc:492] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: No bootstrap required, opened a new log
I20260810 23:43:34.560369 32660 ts_tablet_manager.cc:1397] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:43:34.563236 32653 raft_consensus.cc:348] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.563946 32653 raft_consensus.cc:374] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:34.564100 32653 raft_consensus.cc:729] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a0e7f90b234c9eaa5991aa56f0c3ad, State: Initialized, Role: FOLLOWER
I20260810 23:43:34.564337 32607 heartbeater.cc:503] Master 127.30.33.62:40289 was elected leader, sending a full tablet report...
I20260810 23:43:34.564924 32653 consensus_queue.cc:260] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.565196 32651 ts_tablet_manager.cc:1428] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Time spent starting tablet: real 0.018s user 0.009s sys 0.010s
I20260810 23:43:34.566181 32350 heartbeater.cc:503] Master 127.30.33.62:40289 was elected leader, sending a full tablet report...
I20260810 23:43:34.567255 32653 ts_tablet_manager.cc:1428] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260810 23:43:34.570113 32660 raft_consensus.cc:348] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.570749 32660 raft_consensus.cc:374] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:34.570873 32660 raft_consensus.cc:729] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa8430ea8434bf4b9b01793b09da8f8, State: Initialized, Role: FOLLOWER
I20260810 23:43:34.571578 32660 consensus_queue.cc:260] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.573295 32660 ts_tablet_manager.cc:1428] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260810 23:43:34.644371 32668 raft_consensus.cc:482] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:43:34.644722 32668 raft_consensus.cc:504] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.646545 32668 leader_election.cc:288] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4a0e7f90b234c9eaa5991aa56f0c3ad (127.30.33.2:44307), 4fa8430ea8434bf4b9b01793b09da8f8 (127.30.33.1:37857)
I20260810 23:43:34.652882 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.646989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:44307 (local address 127.30.33.3:43599)
0810 23:43:34.647210 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.647231 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:34.647383 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:34.647833 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:34.647838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:34.647868 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:34.648089 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.648101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.649424 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.649430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.651075 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.651105 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.652286 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.652290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.652437 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.652464 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:43:34.652536 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:43:34.652648 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 23:43:34.652997 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.646999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:43599 (local address 127.30.33.2:44307)
0810 23:43:34.647173 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.647178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.647306 (+ 128us) server_negotiation.cc:407] Connection header received
0810 23:43:34.647587 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.647591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.647659 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.647762 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:34.648310 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.649241 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.651278 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.652131 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.652272 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.652374 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.652590 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.652730 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:43:34.654124 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.647862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:47189 (local address 127.30.33.1:37857)
0810 23:43:34.648094 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.648100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.648481 (+ 381us) server_negotiation.cc:407] Connection header received
0810 23:43:34.648593 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.648599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.648701 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.648811 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:34.649430 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.650127 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.652149 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.653318 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.653451 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.653549 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.653846 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.653947 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:43:34.654210 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.647502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.30.33.3:47189)
0810 23:43:34.648371 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.648386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:34.648483 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:34.648914 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:34.648917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:34.648942 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:34.649240 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.649248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.650292 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.650298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.651950 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:34.651963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.653476 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.653480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:34.653668 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.653703 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:43:34.653797 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:43:34.653925 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":540,"threads_started":1}
I20260810 23:43:34.654119 32274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b585059013c3488eb152b2a4ff743313" candidate_uuid: "daf1e1d22e454edb84684da211dea728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" is_pre_election: true
I20260810 23:43:34.654743 32274 raft_consensus.cc:2454] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daf1e1d22e454edb84684da211dea728 in term 0.
I20260810 23:43:34.655252 32044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b585059013c3488eb152b2a4ff743313" candidate_uuid: "daf1e1d22e454edb84684da211dea728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" is_pre_election: true
I20260810 23:43:34.655488 32386 leader_election.cc:302] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4a0e7f90b234c9eaa5991aa56f0c3ad, daf1e1d22e454edb84684da211dea728; no voters:
I20260810 23:43:34.655902 32044 raft_consensus.cc:2454] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daf1e1d22e454edb84684da211dea728 in term 0.
I20260810 23:43:34.655962 32668 raft_consensus.cc:2781] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:43:34.656101 32668 raft_consensus.cc:482] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:43:34.656200 32668 raft_consensus.cc:3037] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:34.658528 32668 raft_consensus.cc:504] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.659155 32668 leader_election.cc:288] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [CANDIDATE]: Term 1 election: Requested vote from peers c4a0e7f90b234c9eaa5991aa56f0c3ad (127.30.33.2:44307), 4fa8430ea8434bf4b9b01793b09da8f8 (127.30.33.1:37857)
I20260810 23:43:34.659750 32274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b585059013c3488eb152b2a4ff743313" candidate_uuid: "daf1e1d22e454edb84684da211dea728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad"
I20260810 23:43:34.659787 32044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b585059013c3488eb152b2a4ff743313" candidate_uuid: "daf1e1d22e454edb84684da211dea728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa8430ea8434bf4b9b01793b09da8f8"
I20260810 23:43:34.659955 32274 raft_consensus.cc:3037] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:34.660012 32044 raft_consensus.cc:3037] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:34.662542 32274 raft_consensus.cc:2454] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daf1e1d22e454edb84684da211dea728 in term 1.
I20260810 23:43:34.662621 32044 raft_consensus.cc:2454] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daf1e1d22e454edb84684da211dea728 in term 1.
I20260810 23:43:34.662966 32386 leader_election.cc:302] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4a0e7f90b234c9eaa5991aa56f0c3ad, daf1e1d22e454edb84684da211dea728; no voters:
I20260810 23:43:34.663281 32668 raft_consensus.cc:2781] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:34.663770 32668 raft_consensus.cc:686] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 1 LEADER]: Becoming Leader. State: Replica: daf1e1d22e454edb84684da211dea728, State: Running, Role: LEADER
I20260810 23:43:34.664350 32668 consensus_queue.cc:237] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } }
I20260810 23:43:34.669234 31807 catalog_manager.cc:5495] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 reported cstate change: term changed from 0 to 1, leader changed from <none> to daf1e1d22e454edb84684da211dea728 (127.30.33.3). New cstate: current_term: 1 leader_uuid: "daf1e1d22e454edb84684da211dea728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf1e1d22e454edb84684da211dea728" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 40163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 } health_report { overall_health: UNKNOWN } } }
W20260810 23:43:34.695251 32608 tablet.cc:2369] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:43:34.706252 32353 tablet.cc:2369] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:34.711697 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.705854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.30.33.3:40163)
0810 23:43:34.706057 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.706063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.706082 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:43:34.706326 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.706331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.706451 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.706597 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:43:34.707031 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.707879 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.709381 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.709814 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.709930 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.710497 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:43:34.711050 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:43:34.711200 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.711324 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.711413 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:43:34.720252 32044 raft_consensus.cc:1261] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 1 FOLLOWER]: Refusing update from remote peer daf1e1d22e454edb84684da211dea728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:34.720264 32274 raft_consensus.cc:1261] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 1 FOLLOWER]: Refusing update from remote peer daf1e1d22e454edb84684da211dea728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:34.721222 32675 consensus_queue.cc:1035] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa8430ea8434bf4b9b01793b09da8f8" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:34.721583 32668 consensus_queue.cc:1035] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a0e7f90b234c9eaa5991aa56f0c3ad" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 44307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:43:34.728844 32127 tablet.cc:2369] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:34.748199 31807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:34.748502 31807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:34.752398 31807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cfccf07bba74e7c8bfc64f9f8af7df4: Sending DeleteTablet for 3 replicas of tablet b585059013c3488eb152b2a4ff743313
I20260810 23:43:34.753762 32478 tablet_service.cc:1514] Processing DeleteTablet for tablet b585059013c3488eb152b2a4ff743313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:34 UTC) from {username='slave'} at 127.0.0.1:58584
I20260810 23:43:34.753913 32242 tablet_service.cc:1514] Processing DeleteTablet for tablet b585059013c3488eb152b2a4ff743313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:34 UTC) from {username='slave'} at 127.0.0.1:50318
I20260810 23:43:34.753995 32011 tablet_service.cc:1514] Processing DeleteTablet for tablet b585059013c3488eb152b2a4ff743313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:34 UTC) from {username='slave'} at 127.0.0.1:43668
I20260810 23:43:34.756038 32696 tablet_replica.cc:330] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: stopping tablet replica
I20260810 23:43:34.756052 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31910
I20260810 23:43:34.756701 32697 tablet_replica.cc:330] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: stopping tablet replica
I20260810 23:43:34.757190 32697 raft_consensus.cc:2229] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:34.757761 32697 raft_consensus.cc:2258] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:34.756537 32696 raft_consensus.cc:2229] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:34.758627 32696 raft_consensus.cc:2258] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:34.759438 32694 tablet_replica.cc:330] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: stopping tablet replica
I20260810 23:43:34.760022 32694 raft_consensus.cc:2229] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:43:34.760805 32694 raft_consensus.cc:2258] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:34.761823 32696 ts_tablet_manager.cc:1905] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:34.763427 32697 ts_tablet_manager.cc:1905] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:34.769639 32696 ts_tablet_manager.cc:1918] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:34.769930 32696 log.cc:1196] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/b585059013c3488eb152b2a4ff743313
I20260810 23:43:34.770048 32694 ts_tablet_manager.cc:1905] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:34.770599 32696 ts_tablet_manager.cc:1939] T b585059013c3488eb152b2a4ff743313 P c4a0e7f90b234c9eaa5991aa56f0c3ad: Deleting consensus metadata
I20260810 23:43:34.771034 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.758944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.30.33.1:37857)
0810 23:43:34.759247 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.759253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.761783 (+ 2530us) server_negotiation.cc:407] Connection header received
0810 23:43:34.761974 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.761980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.762083 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.762208 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:34.762884 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.763981 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.766064 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.766747 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.766850 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.769158 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:34.769177 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:34.769198 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:34.770010 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:34.770079 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:34.770109 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:34.770209 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.770499 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.770629 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260810 23:43:34.771662 32697 ts_tablet_manager.cc:1918] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:34.771919 32697 log.cc:1196] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/b585059013c3488eb152b2a4ff743313
I20260810 23:43:34.771984 32120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:34.772755 31777 catalog_manager.cc:4841] TS c4a0e7f90b234c9eaa5991aa56f0c3ad (127.30.33.2:44307): tablet b585059013c3488eb152b2a4ff743313 (table test-table [id=d34a1ebff4d14905a47987d9bb6b8dcb]) successfully deleted
I20260810 23:43:34.777887 32694 ts_tablet_manager.cc:1918] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:34.778121 32694 log.cc:1196] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/b585059013c3488eb152b2a4ff743313
I20260810 23:43:34.778723 32694 ts_tablet_manager.cc:1939] T b585059013c3488eb152b2a4ff743313 P daf1e1d22e454edb84684da211dea728: Deleting consensus metadata
I20260810 23:43:34.780995 31781 catalog_manager.cc:4841] TS daf1e1d22e454edb84684da211dea728 (127.30.33.3:40163): tablet b585059013c3488eb152b2a4ff743313 (table test-table [id=d34a1ebff4d14905a47987d9bb6b8dcb]) successfully deleted
I20260810 23:43:34.875111 32697 ts_tablet_manager.cc:1939] T b585059013c3488eb152b2a4ff743313 P 4fa8430ea8434bf4b9b01793b09da8f8: Deleting consensus metadata
I20260810 23:43:34.875368 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31910
W20260810 23:43:34.903159 31777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.33.1:37857 (error 108)
I20260810 23:43:34.903733 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 32132
I20260810 23:43:34.905556 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.904986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43680)
0810 23:43:34.905236 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.905346 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260810 23:43:34.905879 31777 catalog_manager.cc:4584] TS 4fa8430ea8434bf4b9b01793b09da8f8 (127.30.33.1:37857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b585059013c3488eb152b2a4ff743313: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
I20260810 23:43:34.912662 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.904681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.30.33.2:44307)
0810 23:43:34.904952 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:34.904959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:34.904982 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:43:34.905091 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:34.905097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:34.905246 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:34.905365 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:34.905946 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.907137 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.908945 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:34.909653 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:34.909796 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:34.910897 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:34.910911 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:34.910932 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:34.911810 (+ 878us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:34.911883 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:34.911913 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:34.912045 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:34.912129 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 23:43:34.912271 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260810 23:43:34.913497 32342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:34.959945 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.959330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43686)
0810 23:43:34.959639 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.959756 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260810 23:43:34.961704 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:34.961195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43698)
0810 23:43:34.961400 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:34.961508 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:43:35.005008 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 32132
I20260810 23:43:35.022668 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:35.022083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43706)
0810 23:43:35.022372 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:35.022488 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 23:43:35.024046 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:35.023609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43708)
0810 23:43:35.023784 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:35.023883 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:43:35.036420 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 32362
I20260810 23:43:35.044176 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:35.037220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.30.33.3:40163)
0810 23:43:35.037426 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:35.037433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:35.037513 (+ 80us) server_negotiation.cc:407] Connection header received
0810 23:43:35.037723 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:35.037727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:35.037839 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:35.037930 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:35.038429 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:35.039267 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:35.040814 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:35.041339 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:35.041467 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:35.042651 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:35.042662 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:35.042680 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:35.043341 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:35.043392 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:35.043419 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:35.043579 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:35.043704 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 23:43:35.043831 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:43:35.044978 32593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:35.105074 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:35.104504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43716)
0810 23:43:35.104758 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:35.104880 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 23:43:35.106443 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:35.105863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43730)
0810 23:43:35.106099 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:35.106215 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 23:43:35.171371 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 32362
I20260810 23:43:35.201789 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 31746
I20260810 23:43:35.202996 31855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:35.314368 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:35.313862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:37857 (local address 127.0.0.1:43732)
0810 23:43:35.314068 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:35.314181 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:43:35.314541 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 31746
2026-08-10T23:43:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3065 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:43:35 dist-test-slave-pww0 krb5kdc[32711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:43:35 dist-test-slave-pww0 krb5kdc[32711](info): set up 2 sockets
Aug 10 23:43:35 dist-test-slave-pww0 krb5kdc[32711](info): commencing operation
krb5kdc: starting...
W20260810 23:43:36.638581 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:43:36 dist-test-slave-pww0 krb5kdc[32711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:43:36.973129 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:33093
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:33093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:37.337631 32746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:37.337993 32746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:37.338145 32746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:37.345860 32746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:37.346014 32746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:37.346099 32746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:37.346180 32746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:37.346243 32746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:37.356750 32746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:33093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:33093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:37.358223 32746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:37.359639 32746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:37.372187 32759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:37.372303 32755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:37.372423 32754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:37.373044 32746 server_base.cc:1034] running on GCE node
I20260810 23:43:37.374040 32746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:37.375763 32746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:37.377012 32746 hybrid_clock.cc:648] HybridClock initialized: now 1786405417376950 us; error 56 us; skew 500 ppm
I20260810 23:43:37.377421 32746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:37 dist-test-slave-pww0 krb5kdc[32711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405417, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:37.380885 32746 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:43:37.382616 32746 webserver.cc:460] Webserver started at http://127.30.33.62:34247/ using document root <none> and password file <none>
I20260810 23:43:37.383397 32746 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:37.383567 32746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:37.383877 32746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:37.387145 32746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "dc9b2ade513e47b0910309828d3f0979"
format_stamp: "Formatted at 2026-08-10 23:43:37 on dist-test-slave-pww0"
server_key: "16396ae80cef2cd2027956057d46440e"
server_key_iv: "51a3a323796b9c3f6d9214940a1d5f6a"
server_key_version: "encryptionkey@0"
I20260810 23:43:37.388140 32746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "dc9b2ade513e47b0910309828d3f0979"
format_stamp: "Formatted at 2026-08-10 23:43:37 on dist-test-slave-pww0"
server_key: "16396ae80cef2cd2027956057d46440e"
server_key_iv: "51a3a323796b9c3f6d9214940a1d5f6a"
server_key_version: "encryptionkey@0"
I20260810 23:43:37.394009 32746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:43:37.397921 305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:37.399616 32746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:37.399827 32746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "dc9b2ade513e47b0910309828d3f0979"
format_stamp: "Formatted at 2026-08-10 23:43:37 on dist-test-slave-pww0"
server_key: "16396ae80cef2cd2027956057d46440e"
server_key_iv: "51a3a323796b9c3f6d9214940a1d5f6a"
server_key_version: "encryptionkey@0"
I20260810 23:43:37.400055 32746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:37.421309 32746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:37.426316 32746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:37.426611 32746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:37.447513 32746 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:33093
I20260810 23:43:37.447567 412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:33093 every 8 connection(s)
I20260810 23:43:37.449155 32746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:37.453178 413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:37.457731 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 32746
I20260810 23:43:37.457949 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:37.458508 30852 external_mini_cluster.cc:1428] Setting key 3c1340c226c506f828537c2f576c6e24
I20260810 23:43:37.465583 413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979: Bootstrap starting.
I20260810 23:43:37.469794 413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:37.471148 413 log.cc:824] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:37.475015 413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979: No bootstrap required, opened a new log
Aug 10 23:43:37 dist-test-slave-pww0 krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:37.483144 413 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b2ade513e47b0910309828d3f0979" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33093 } }
I20260810 23:43:37.483669 413 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:37.483812 413 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9b2ade513e47b0910309828d3f0979, State: Initialized, Role: FOLLOWER
I20260810 23:43:37.484412 413 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b2ade513e47b0910309828d3f0979" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33093 } }
I20260810 23:43:37.484666 413 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:37.484805 413 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:37.484984 413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:37.485040 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:37.460659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.30.33.62:33093)
0810 23:43:37.461547 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:37.461560 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:37.461610 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:43:37.463897 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:37.463928 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:37.464435 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:37.464820 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:37.466939 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:37.468461 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:37.470544 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:37.471358 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:37.471574 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:37.476774 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:37.476784 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:37.476804 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:37.476853 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:37.480196 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:37.481047 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:37.481055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:37.481063 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:37.481276 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:37.481711 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:37.481715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:37.481717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:37.482365 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:37.482660 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:37.483198 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 23:43:37.483746 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":403,"threads_started":1}
I20260810 23:43:37.487942 413 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b2ade513e47b0910309828d3f0979" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33093 } }
I20260810 23:43:37.488548 413 leader_election.cc:302] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc9b2ade513e47b0910309828d3f0979; no voters:
I20260810 23:43:37.489125 413 leader_election.cc:288] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:37.489251 422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:37.489910 422 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [term 1 LEADER]: Becoming Leader. State: Replica: dc9b2ade513e47b0910309828d3f0979, State: Running, Role: LEADER
I20260810 23:43:37.490446 422 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b2ade513e47b0910309828d3f0979" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33093 } }
I20260810 23:43:37.491180 413 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:37.493779 428 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9b2ade513e47b0910309828d3f0979. Latest consensus state: current_term: 1 leader_uuid: "dc9b2ade513e47b0910309828d3f0979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b2ade513e47b0910309828d3f0979" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33093 } } }
I20260810 23:43:37.493886 423 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc9b2ade513e47b0910309828d3f0979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9b2ade513e47b0910309828d3f0979" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33093 } } }
I20260810 23:43:37.494275 428 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:37.494282 423 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:37.496477 433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:37.501578 433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:37.511924 433 catalog_manager.cc:1347] Generated new cluster ID: 8e7c556c522648738f10385b678b7562
I20260810 23:43:37.512187 433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:37.539507 433 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:37.540951 433 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:37.549472 433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979: Generated new TSK 0
I20260810 23:43:37.550448 433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:43:37.628836 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33093
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:37.990206 453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:37.990589 453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:37.990715 453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:37.998688 453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:37.998829 453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:37.998950 453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:43:38.009932 453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:38.011528 453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:38.013039 453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:38.026723 468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:38.026830 465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:38.026798 470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:38.027719 453 server_base.cc:1034] running on GCE node
I20260810 23:43:38.028409 453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:38.029445 453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:38.030668 453 hybrid_clock.cc:648] HybridClock initialized: now 1786405418030602 us; error 61 us; skew 500 ppm
I20260810 23:43:38.030998 453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:38 dist-test-slave-pww0 krb5kdc[32711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405418, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:38.037932 453 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:43:38.039594 453 webserver.cc:460] Webserver started at http://127.30.33.1:42225/ using document root <none> and password file <none>
I20260810 23:43:38.040341 453 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:38.040475 453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:38.040813 453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:38.044395 453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95"
format_stamp: "Formatted at 2026-08-10 23:43:38 on dist-test-slave-pww0"
server_key: "ba95c2fdf059a585b178afb38df68f34"
server_key_iv: "a34e5dc616f14853376770deb1f9d843"
server_key_version: "encryptionkey@0"
I20260810 23:43:38.045469 453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95"
format_stamp: "Formatted at 2026-08-10 23:43:38 on dist-test-slave-pww0"
server_key: "ba95c2fdf059a585b178afb38df68f34"
server_key_iv: "a34e5dc616f14853376770deb1f9d843"
server_key_version: "encryptionkey@0"
I20260810 23:43:38.051486 453 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:43:38.055272 486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:38.056924 453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:38.057118 453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95"
format_stamp: "Formatted at 2026-08-10 23:43:38 on dist-test-slave-pww0"
server_key: "ba95c2fdf059a585b178afb38df68f34"
server_key_iv: "a34e5dc616f14853376770deb1f9d843"
server_key_version: "encryptionkey@0"
I20260810 23:43:38.057354 453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:38.096971 453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:38.102150 453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:38.102455 453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:38.103611 453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:38.105161 453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:38.105254 453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:38.105389 453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:38.105450 453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:38.137233 453 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43239
I20260810 23:43:38.137307 716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43239 every 8 connection(s)
I20260810 23:43:38.138604 453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:43:38.143536 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 453
I20260810 23:43:38.143748 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:43:38.144182 30852 external_mini_cluster.cc:1428] Setting key 90bfe8d7da738faf9b528599a7dca51e
Aug 10 23:43:38 dist-test-slave-pww0 krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405418, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:38.159839 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:38.141379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:58279 (local address 127.30.33.62:33093)
0810 23:43:38.141852 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:38.141856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:38.142761 (+ 905us) server_negotiation.cc:407] Connection header received
0810 23:43:38.144849 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:38.144855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:38.144949 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:38.145051 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:38.146813 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.147604 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.149247 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.149668 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.149801 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:38.153793 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:38.153798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:38.153801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:38.153840 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:38.156494 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:38.157332 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:38.157336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:38.157339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:38.157473 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:38.157935 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:38.157939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:38.157941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:38.158237 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:38.158380 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:38.159163 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 23:43:38.159435 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":235,"threads_started":1}
I20260810 23:43:38.160907 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:38.141801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33093 (local address 127.30.33.1:58279)
0810 23:43:38.142553 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:38.142601 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:38.144524 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:38.145336 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:38.145345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:38.145912 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:38.146619 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:38.146640 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.147779 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.147783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:38.149086 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:38.149100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.149924 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.149931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:38.150180 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:38.150896 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:38.150920 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:38.153497 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:38.156670 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:38.156681 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:38.156697 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:38.157182 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:38.157592 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:38.157597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:38.157600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:38.157814 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:38.158373 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:38.158382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:38.158756 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 23:43:38.159026 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 23:43:38.159676 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":291,"threads_started":1}
I20260810 23:43:38.162667 717 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33093
I20260810 23:43:38.163012 717 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:38.163829 717 heartbeater.cc:511] Master 127.30.33.62:33093 requested a full tablet report, sending...
I20260810 23:43:38.166361 339 ts_manager.cc:194] Registered new tserver with Master: 5fa1bdf7e3fc4bfb8743ae4da4119c95 (127.30.33.1:43239)
I20260810 23:43:38.168525 339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:58279
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:43:38.216620 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33093
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:38.570012 726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:38.570309 726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:38.570456 726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:38.577798 726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:38.577916 726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:38.578075 726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:43:38.588155 726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:38.589573 726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:38.591012 726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:38.604513 738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:38.604532 740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:38.604583 745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:38.605847 726 server_base.cc:1034] running on GCE node
I20260810 23:43:38.606526 726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:38.607661 726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:38.608913 726 hybrid_clock.cc:648] HybridClock initialized: now 1786405418608855 us; error 45 us; skew 500 ppm
I20260810 23:43:38.609266 726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:38 dist-test-slave-pww0 krb5kdc[32711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405418, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:38.612650 726 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:43:38.614203 726 webserver.cc:460] Webserver started at http://127.30.33.2:46303/ using document root <none> and password file <none>
I20260810 23:43:38.614900 726 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:38.615006 726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:38.615312 726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:38.618363 726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "60623b5882334632a94ea086b83a3f72"
format_stamp: "Formatted at 2026-08-10 23:43:38 on dist-test-slave-pww0"
server_key: "905833f586c25bf4c6786e1808268833"
server_key_iv: "3c1f0de8904bcbca9bd439583f572d99"
server_key_version: "encryptionkey@0"
I20260810 23:43:38.619122 726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "60623b5882334632a94ea086b83a3f72"
format_stamp: "Formatted at 2026-08-10 23:43:38 on dist-test-slave-pww0"
server_key: "905833f586c25bf4c6786e1808268833"
server_key_iv: "3c1f0de8904bcbca9bd439583f572d99"
server_key_version: "encryptionkey@0"
I20260810 23:43:38.624564 726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:43:38.628201 763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:38.629630 726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:38.629786 726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "60623b5882334632a94ea086b83a3f72"
format_stamp: "Formatted at 2026-08-10 23:43:38 on dist-test-slave-pww0"
server_key: "905833f586c25bf4c6786e1808268833"
server_key_iv: "3c1f0de8904bcbca9bd439583f572d99"
server_key_version: "encryptionkey@0"
I20260810 23:43:38.629979 726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:38.652120 726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:38.656769 726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:38.657042 726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:38.658046 726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:38.659493 726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:38.659571 726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:38.659667 726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:38.659709 726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:38.691871 726 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:43375
I20260810 23:43:38.691917 963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:43375 every 8 connection(s)
I20260810 23:43:38.693199 726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:43:38.700219 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 726
I20260810 23:43:38.700407 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:43:38.700875 30852 external_mini_cluster.cc:1428] Setting key ba7219dface871deec524432220ca219
Aug 10 23:43:38 dist-test-slave-pww0 krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405418, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:38.714388 973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:38.695742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:59599 (local address 127.30.33.62:33093)
0810 23:43:38.696177 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:38.696182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:38.697150 (+ 968us) server_negotiation.cc:407] Connection header received
0810 23:43:38.699106 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:38.699112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:38.699201 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:38.699302 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:38.701166 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.702022 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.703900 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.704333 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.704452 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:38.708380 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:38.708386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:38.708389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:38.708427 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:38.710968 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:38.711961 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:38.711966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:38.711968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:38.712130 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:38.712633 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:38.712637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:38.712639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:38.712927 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:38.713042 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:38.713921 (+ 879us) server_negotiation.cc:299] Negotiation successful
0810 23:43:38.714120 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":215,"threads_started":1}
I20260810 23:43:38.715883 974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:38.696252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33093 (local address 127.30.33.2:59599)
0810 23:43:38.696971 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:38.697025 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:38.698813 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:38.699628 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:38.699637 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:38.700247 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:38.700985 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:38.701013 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.702190 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.702196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:38.703750 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:38.703767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:38.704568 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:38.704576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:38.704841 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:38.705542 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:38.705566 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:38.708110 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:38.711136 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:38.711145 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:38.711165 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:38.711825 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:38.712257 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:38.712261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:38.712264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:38.712527 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:38.713057 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:38.713065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:38.713532 (+ 467us) client_negotiation.cc:769] Sending connection context
0810 23:43:38.713842 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 23:43:38.714429 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":271,"threads_started":1}
I20260810 23:43:38.717475 969 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33093
I20260810 23:43:38.717797 969 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:38.718551 969 heartbeater.cc:511] Master 127.30.33.62:33093 requested a full tablet report, sending...
I20260810 23:43:38.720288 339 ts_manager.cc:194] Registered new tserver with Master: 60623b5882334632a94ea086b83a3f72 (127.30.33.2:43375)
I20260810 23:43:38.721139 339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:59599
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:43:38.772523 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33093
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:39.127753 978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:39.128206 978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:39.128376 978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:39.137995 978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:39.138152 978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:39.138350 978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:43:39.149153 978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:39.150643 978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:39.152148 978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:39.165920 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:39.165962 992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:39.165987 999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:39.167207 978 server_base.cc:1034] running on GCE node
I20260810 23:43:39.167929 978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:39.168980 978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:39.170214 978 hybrid_clock.cc:648] HybridClock initialized: now 1786405419170153 us; error 75 us; skew 500 ppm
I20260810 23:43:39.170570 978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:39.172137 717 heartbeater.cc:503] Master 127.30.33.62:33093 was elected leader, sending a full tablet report...
Aug 10 23:43:39 dist-test-slave-pww0 krb5kdc[32711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405419, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:39.174275 978 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:43:39.175967 978 webserver.cc:460] Webserver started at http://127.30.33.3:33261/ using document root <none> and password file <none>
I20260810 23:43:39.176677 978 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:39.176792 978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:39.177088 978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:39.180370 978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "eb54158263a54a8b97662cd2c42c730b"
format_stamp: "Formatted at 2026-08-10 23:43:39 on dist-test-slave-pww0"
server_key: "88c42bc83edab0e1b87b2e2c643f9e79"
server_key_iv: "f14a7deca27d93937d689efb2665d53b"
server_key_version: "encryptionkey@0"
I20260810 23:43:39.181144 978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "eb54158263a54a8b97662cd2c42c730b"
format_stamp: "Formatted at 2026-08-10 23:43:39 on dist-test-slave-pww0"
server_key: "88c42bc83edab0e1b87b2e2c643f9e79"
server_key_iv: "f14a7deca27d93937d689efb2665d53b"
server_key_version: "encryptionkey@0"
I20260810 23:43:39.186765 978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:43:39.190558 1021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:39.192227 978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:43:39.192420 978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "eb54158263a54a8b97662cd2c42c730b"
format_stamp: "Formatted at 2026-08-10 23:43:39 on dist-test-slave-pww0"
server_key: "88c42bc83edab0e1b87b2e2c643f9e79"
server_key_iv: "f14a7deca27d93937d689efb2665d53b"
server_key_version: "encryptionkey@0"
I20260810 23:43:39.192633 978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:39.235682 978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:39.240340 978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:39.240657 978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:39.241695 978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:39.243115 978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:39.243194 978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:39.243312 978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:39.243363 978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:39.275099 978 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:35511
I20260810 23:43:39.275175 1253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:35511 every 8 connection(s)
I20260810 23:43:39.276521 978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:43:39.277541 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 978
I20260810 23:43:39.277735 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:43:39.278287 30852 external_mini_cluster.cc:1428] Setting key a2ee01e214f09acb925104064e15b453
Aug 10 23:43:39 dist-test-slave-pww0 krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405419, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:39.298236 1257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.279537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:46541 (local address 127.30.33.62:33093)
0810 23:43:39.280027 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.280033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.281194 (+ 1161us) server_negotiation.cc:407] Connection header received
0810 23:43:39.283178 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.283183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.283272 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.283366 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:39.285030 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.285850 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.287511 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.287918 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.288051 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.291680 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:39.291685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:39.291689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:39.291726 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:39.294901 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.295743 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.295747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.295750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.295872 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.296367 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.296371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.296373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.296688 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:39.296858 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.297620 (+ 762us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.297857 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260810 23:43:39.299329 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.280246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33093 (local address 127.30.33.3:46541)
0810 23:43:39.281036 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:39.281092 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:39.282896 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:39.283658 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:39.283667 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:39.284228 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:39.284851 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.284872 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.286016 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.286021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.287323 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.287342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.288172 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.288180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.288415 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.289096 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:39.289120 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:39.291417 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:39.295052 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:39.295062 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:39.295081 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:39.295618 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:39.295980 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:39.295986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:39.295990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:39.296225 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:39.296834 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:39.296842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:39.297226 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 23:43:39.297477 (+ 251us) client_negotiation.cc:240] Negotiation successful
0810 23:43:39.298097 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":314,"threads_started":1}
I20260810 23:43:39.300778 1254 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33093
I20260810 23:43:39.301107 1254 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:39.301846 1254 heartbeater.cc:511] Master 127.30.33.62:33093 requested a full tablet report, sending...
I20260810 23:43:39.303423 339 ts_manager.cc:194] Registered new tserver with Master: eb54158263a54a8b97662cd2c42c730b (127.30.33.3:35511)
I20260810 23:43:39.304423 339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:46541
I20260810 23:43:39.307224 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:43:39.325274 1257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.311675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.30.33.62:33093)
0810 23:43:39.311860 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.311864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.312272 (+ 408us) server_negotiation.cc:407] Connection header received
0810 23:43:39.312425 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.312429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.312496 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.312583 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:39.314031 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.314764 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.316142 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.316547 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.316644 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.319934 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:39.319939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:39.319942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:39.319980 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:39.322557 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.323385 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.323389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.323391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.323542 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.323957 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.323960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.323962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.324276 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:39.324488 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.324889 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.325037 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:43:39.329589 339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:43:39.336844 339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:43:39.361984 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.350230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:43375 (local address 127.0.0.1:60858)
0810 23:43:39.352475 (+ 2245us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:39.352515 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:39.352696 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:39.353923 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:39.353932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:39.353966 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:39.354437 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.354453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.356448 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.356453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.358921 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.358938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.360882 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.360888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.361082 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.361137 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 23:43:39.361320 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 23:43:39.361532 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1967,"mutex_wait_us":83,"thread_start_us":2732,"threads_started":1}
I20260810 23:43:39.362985 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.350296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.30.33.2:43375)
0810 23:43:39.353176 (+ 2880us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.353185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.353217 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:43:39.353347 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.353356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.353637 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.354644 (+ 1007us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:39.354921 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.356309 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.359121 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.360677 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.360819 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.361027 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.361379 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.361563 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2713,"thread_start_us":3056,"threads_started":1}
I20260810 23:43:39.363025 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.353033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43239 (local address 127.0.0.1:49932)
0810 23:43:39.354038 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:39.354065 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:39.354206 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:39.355405 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:39.355410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:39.355436 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:39.355921 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.355935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.357864 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.357870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.359910 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.359927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.361818 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.361824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.362283 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.362328 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:43:39.362396 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:43:39.362588 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":75,"thread_start_us":786,"threads_started":1}
I20260810 23:43:39.363592 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.350340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:35511 (local address 127.0.0.1:36342)
0810 23:43:39.353279 (+ 2939us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:39.353296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:39.354841 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:39.355710 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:39.355713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:39.355765 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:39.356119 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.356130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.357668 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.357674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.359884 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.359904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.362194 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.362201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.362381 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.362414 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:43:39.362492 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:43:39.362615 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2823,"mutex_wait_us":33,"thread_start_us":1734,"threads_started":1}
I20260810 23:43:39.363611 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.350946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.30.33.3:35511)
0810 23:43:39.351907 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.351918 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.354780 (+ 2862us) server_negotiation.cc:407] Connection header received
0810 23:43:39.355017 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.355024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.355203 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.355402 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:39.356310 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.357492 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.360106 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.361663 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.361802 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.362037 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.363059 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.363252 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":621,"threads_started":1}
I20260810 23:43:39.364065 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.353622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.30.33.1:43239)
0810 23:43:39.354788 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.354796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.354820 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:43:39.354931 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.354939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.355193 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.355402 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:39.356137 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.357677 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.360110 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.361583 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.363262 (+ 1679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.363497 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.363590 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.363784 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":871,"threads_started":1}
I20260810 23:43:39.367061 855 tablet_service.cc:1467] Processing CreateTablet for tablet 1865b2e02c97465ebd508f09f1c4d83b (DEFAULT_TABLE table=test-table [id=13b4c30aedcf45419e0b3c64c16631f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:39.367496 580 tablet_service.cc:1467] Processing CreateTablet for tablet 1865b2e02c97465ebd508f09f1c4d83b (DEFAULT_TABLE table=test-table [id=13b4c30aedcf45419e0b3c64c16631f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:39.367496 1121 tablet_service.cc:1467] Processing CreateTablet for tablet 1865b2e02c97465ebd508f09f1c4d83b (DEFAULT_TABLE table=test-table [id=13b4c30aedcf45419e0b3c64c16631f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:39.368798 855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1865b2e02c97465ebd508f09f1c4d83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:39.369045 1121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1865b2e02c97465ebd508f09f1c4d83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:39.369046 580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1865b2e02c97465ebd508f09f1c4d83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:39.381138 1284 tablet_bootstrap.cc:492] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Bootstrap starting.
I20260810 23:43:39.384148 1288 tablet_bootstrap.cc:492] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Bootstrap starting.
I20260810 23:43:39.385823 1289 tablet_bootstrap.cc:492] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Bootstrap starting.
I20260810 23:43:39.386619 1284 tablet_bootstrap.cc:654] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:39.387876 1288 tablet_bootstrap.cc:654] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:39.388051 1284 log.cc:824] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:39.388959 1289 tablet_bootstrap.cc:654] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:39.389216 1288 log.cc:824] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:39.390097 1289 log.cc:824] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:39.392096 1284 tablet_bootstrap.cc:492] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: No bootstrap required, opened a new log
I20260810 23:43:39.392494 1284 ts_tablet_manager.cc:1397] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 23:43:39.396442 1288 tablet_bootstrap.cc:492] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: No bootstrap required, opened a new log
I20260810 23:43:39.396775 1288 ts_tablet_manager.cc:1397] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260810 23:43:39.397078 1289 tablet_bootstrap.cc:492] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: No bootstrap required, opened a new log
I20260810 23:43:39.397392 1289 ts_tablet_manager.cc:1397] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260810 23:43:39.403280 1284 raft_consensus.cc:348] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.403903 1284 raft_consensus.cc:374] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:39.404019 1284 raft_consensus.cc:729] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60623b5882334632a94ea086b83a3f72, State: Initialized, Role: FOLLOWER
I20260810 23:43:39.404728 1284 consensus_queue.cc:260] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.405483 1288 raft_consensus.cc:348] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.405860 1288 raft_consensus.cc:374] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:39.405962 1288 raft_consensus.cc:729] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb54158263a54a8b97662cd2c42c730b, State: Initialized, Role: FOLLOWER
I20260810 23:43:39.406589 1288 consensus_queue.cc:260] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.407727 1254 heartbeater.cc:503] Master 127.30.33.62:33093 was elected leader, sending a full tablet report...
I20260810 23:43:39.408486 1288 ts_tablet_manager.cc:1428] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260810 23:43:39.408632 1289 raft_consensus.cc:348] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.409277 1289 raft_consensus.cc:374] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:39.409487 1289 raft_consensus.cc:729] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa1bdf7e3fc4bfb8743ae4da4119c95, State: Initialized, Role: FOLLOWER
I20260810 23:43:39.410276 1289 consensus_queue.cc:260] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.412340 1289 ts_tablet_manager.cc:1428] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260810 23:43:39.412477 969 heartbeater.cc:503] Master 127.30.33.62:33093 was elected leader, sending a full tablet report...
I20260810 23:43:39.413910 1284 ts_tablet_manager.cc:1428] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
W20260810 23:43:39.445333 970 tablet.cc:2369] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:43:39.528403 1256 tablet.cc:2369] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:39.635582 1298 raft_consensus.cc:482] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:43:39.635891 1298 raft_consensus.cc:504] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.637679 1298 leader_election.cc:288] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60623b5882334632a94ea086b83a3f72 (127.30.33.2:43375), 5fa1bdf7e3fc4bfb8743ae4da4119c95 (127.30.33.1:43239)
W20260810 23:43:39.641255 719 tablet.cc:2369] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:39.644701 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.638358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:33785 (local address 127.30.33.2:43375)
0810 23:43:39.638623 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.638629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.638652 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:43:39.638899 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.638903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.638967 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.639045 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:39.639540 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.640539 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.642513 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.643799 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.643937 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.644019 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.644375 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.644497 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 23:43:39.644771 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.638282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:43375 (local address 127.30.33.3:33785)
0810 23:43:39.638546 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:39.638570 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:39.638743 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:39.639094 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:39.639103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:39.639133 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:39.639369 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.639377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.640707 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.640713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.642342 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.642359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.643960 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.643966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.644160 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.644198 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:43:39.644304 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:43:39.644452 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 23:43:39.645955 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.638574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43239 (local address 127.30.33.3:58713)
0810 23:43:39.639336 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:39.639352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:39.639501 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:39.639813 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:39.639818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:39.639844 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:39.640213 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.640225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.641728 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.641733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.643768 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:39.643785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.645349 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.645355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:39.645488 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.645515 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:43:39.645600 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:43:39.645701 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":419,"threads_started":1}
I20260810 23:43:39.646028 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.638382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:58713 (local address 127.30.33.1:43239)
0810 23:43:39.638728 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.638734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.639448 (+ 714us) server_negotiation.cc:407] Connection header received
0810 23:43:39.639628 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.639632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.639703 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.639773 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:39.640385 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.641525 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.643966 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.645127 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.645257 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.645339 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.645655 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.645794 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260810 23:43:39.645916 891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865b2e02c97465ebd508f09f1c4d83b" candidate_uuid: "eb54158263a54a8b97662cd2c42c730b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60623b5882334632a94ea086b83a3f72" is_pre_election: true
I20260810 23:43:39.646600 891 raft_consensus.cc:2454] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb54158263a54a8b97662cd2c42c730b in term 0.
I20260810 23:43:39.647171 626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865b2e02c97465ebd508f09f1c4d83b" candidate_uuid: "eb54158263a54a8b97662cd2c42c730b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" is_pre_election: true
I20260810 23:43:39.647368 1026 leader_election.cc:302] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60623b5882334632a94ea086b83a3f72, eb54158263a54a8b97662cd2c42c730b; no voters:
I20260810 23:43:39.647730 626 raft_consensus.cc:2454] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb54158263a54a8b97662cd2c42c730b in term 0.
I20260810 23:43:39.647874 1298 raft_consensus.cc:2781] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:43:39.648077 1298 raft_consensus.cc:482] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:43:39.648209 1298 raft_consensus.cc:3037] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:39.650756 1298 raft_consensus.cc:504] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.651436 1298 leader_election.cc:288] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [CANDIDATE]: Term 1 election: Requested vote from peers 60623b5882334632a94ea086b83a3f72 (127.30.33.2:43375), 5fa1bdf7e3fc4bfb8743ae4da4119c95 (127.30.33.1:43239)
I20260810 23:43:39.651998 891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865b2e02c97465ebd508f09f1c4d83b" candidate_uuid: "eb54158263a54a8b97662cd2c42c730b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60623b5882334632a94ea086b83a3f72"
I20260810 23:43:39.652055 626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865b2e02c97465ebd508f09f1c4d83b" candidate_uuid: "eb54158263a54a8b97662cd2c42c730b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95"
I20260810 23:43:39.652220 891 raft_consensus.cc:3037] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:39.652251 626 raft_consensus.cc:3037] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:39.654526 891 raft_consensus.cc:2454] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb54158263a54a8b97662cd2c42c730b in term 1.
I20260810 23:43:39.654537 626 raft_consensus.cc:2454] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb54158263a54a8b97662cd2c42c730b in term 1.
I20260810 23:43:39.655021 1026 leader_election.cc:302] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60623b5882334632a94ea086b83a3f72, eb54158263a54a8b97662cd2c42c730b; no voters:
I20260810 23:43:39.655369 1298 raft_consensus.cc:2781] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:39.655850 1298 raft_consensus.cc:686] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 1 LEADER]: Becoming Leader. State: Replica: eb54158263a54a8b97662cd2c42c730b, State: Running, Role: LEADER
I20260810 23:43:39.656344 1298 consensus_queue.cc:237] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } }
I20260810 23:43:39.661563 339 catalog_manager.cc:5495] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b reported cstate change: term changed from 0 to 1, leader changed from <none> to eb54158263a54a8b97662cd2c42c730b (127.30.33.3). New cstate: current_term: 1 leader_uuid: "eb54158263a54a8b97662cd2c42c730b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb54158263a54a8b97662cd2c42c730b" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 35511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 } health_report { overall_health: UNKNOWN } } }
I20260810 23:43:39.732795 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.727127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.30.33.3:35511)
0810 23:43:39.727344 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.727351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.727372 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:43:39.727631 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.727636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.727727 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.727867 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:43:39.728275 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.728989 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.730439 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.730868 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.730982 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.731667 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:43:39.732189 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:43:39.732358 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.732508 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.732567 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:43:39.741828 891 raft_consensus.cc:1261] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 1 FOLLOWER]: Refusing update from remote peer eb54158263a54a8b97662cd2c42c730b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:39.741827 626 raft_consensus.cc:1261] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 1 FOLLOWER]: Refusing update from remote peer eb54158263a54a8b97662cd2c42c730b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:39.742933 1305 consensus_queue.cc:1035] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60623b5882334632a94ea086b83a3f72" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:39.743321 1298 consensus_queue.cc:1035] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa1bdf7e3fc4bfb8743ae4da4119c95" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:39.770118 337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:39.770525 337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:39.774813 337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc9b2ade513e47b0910309828d3f0979: Sending DeleteTablet for 3 replicas of tablet 1865b2e02c97465ebd508f09f1c4d83b
I20260810 23:43:39.776147 1121 tablet_service.cc:1514] Processing DeleteTablet for tablet 1865b2e02c97465ebd508f09f1c4d83b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:39 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:36342
I20260810 23:43:39.776146 580 tablet_service.cc:1514] Processing DeleteTablet for tablet 1865b2e02c97465ebd508f09f1c4d83b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:39 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:49932
I20260810 23:43:39.776206 855 tablet_service.cc:1514] Processing DeleteTablet for tablet 1865b2e02c97465ebd508f09f1c4d83b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:39 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:60858
I20260810 23:43:39.779129 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 453
I20260810 23:43:39.792382 1330 tablet_replica.cc:330] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: stopping tablet replica
I20260810 23:43:39.792615 1328 tablet_replica.cc:330] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: stopping tablet replica
I20260810 23:43:39.793078 1330 raft_consensus.cc:2229] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:39.793282 1329 tablet_replica.cc:330] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: stopping tablet replica
I20260810 23:43:39.793751 1330 raft_consensus.cc:2258] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:39.793751 1329 raft_consensus.cc:2229] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:39.794256 1329 raft_consensus.cc:2258] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:39.794930 1328 raft_consensus.cc:2229] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:43:39.796288 1330 ts_tablet_manager.cc:1905] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:39.796706 1328 raft_consensus.cc:2258] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:39.797231 1329 ts_tablet_manager.cc:1905] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:43:39 dist-test-slave-pww0 krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:43:39.802459 1328 ts_tablet_manager.cc:1905] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:39.803973 1330 ts_tablet_manager.cc:1918] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:39.804255 1330 log.cc:1196] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/1865b2e02c97465ebd508f09f1c4d83b
I20260810 23:43:39.804940 1330 ts_tablet_manager.cc:1939] T 1865b2e02c97465ebd508f09f1c4d83b P 60623b5882334632a94ea086b83a3f72: Deleting consensus metadata
I20260810 23:43:39.804925 1329 ts_tablet_manager.cc:1918] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:39.805105 1329 log.cc:1196] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/1865b2e02c97465ebd508f09f1c4d83b
I20260810 23:43:39.805698 1329 ts_tablet_manager.cc:1939] T 1865b2e02c97465ebd508f09f1c4d83b P 5fa1bdf7e3fc4bfb8743ae4da4119c95: Deleting consensus metadata
I20260810 23:43:39.807174 311 catalog_manager.cc:4841] TS 60623b5882334632a94ea086b83a3f72 (127.30.33.2:43375): tablet 1865b2e02c97465ebd508f09f1c4d83b (table test-table [id=13b4c30aedcf45419e0b3c64c16631f0]) successfully deleted
I20260810 23:43:39.807579 308 catalog_manager.cc:4841] TS 5fa1bdf7e3fc4bfb8743ae4da4119c95 (127.30.33.1:43239): tablet 1865b2e02c97465ebd508f09f1c4d83b (table test-table [id=13b4c30aedcf45419e0b3c64c16631f0]) successfully deleted
I20260810 23:43:39.811298 1328 ts_tablet_manager.cc:1918] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:39.811522 1328 log.cc:1196] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/1865b2e02c97465ebd508f09f1c4d83b
I20260810 23:43:39.812115 1328 ts_tablet_manager.cc:1939] T 1865b2e02c97465ebd508f09f1c4d83b P eb54158263a54a8b97662cd2c42c730b: Deleting consensus metadata
I20260810 23:43:39.813881 307 catalog_manager.cc:4841] TS eb54158263a54a8b97662cd2c42c730b (127.30.33.3:35511): tablet 1865b2e02c97465ebd508f09f1c4d83b (table test-table [id=13b4c30aedcf45419e0b3c64c16631f0]) successfully deleted
I20260810 23:43:39.815678 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.780686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.30.33.1:43239)
0810 23:43:39.780856 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.780862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.786580 (+ 5718us) server_negotiation.cc:407] Connection header received
0810 23:43:39.787576 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.787584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.787739 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.787820 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:39.791772 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.793085 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.794699 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.795212 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.795332 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.808175 (+ 12843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:39.808191 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:39.808211 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:39.808259 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:39.812301 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.813367 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.813379 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.813382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.813561 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.814040 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.814046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.814048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.814464 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:39.814706 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.815110 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.815314 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:43:39.816321 704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:39.917021 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 453
I20260810 23:43:39.941170 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 726
Aug 10 23:43:39 dist-test-slave-pww0 krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:43:39.958163 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:39.942019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.30.33.2:43375)
0810 23:43:39.942215 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:39.942221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:39.942293 (+ 72us) server_negotiation.cc:407] Connection header received
0810 23:43:39.942498 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:39.942503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:39.942609 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:39.942701 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:39.944080 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.944815 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.946167 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:39.946616 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:39.946744 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:39.951790 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:39.951811 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:39.951828 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:39.951872 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:39.954832 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.955658 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.955668 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.955670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.955842 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:39.956310 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:39.956314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:39.956316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:39.956782 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:39.957122 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:39.957546 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 23:43:39.957784 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:43:39.958957 957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:40.078699 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 726
I20260810 23:43:40.108978 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 978
Aug 10 23:43:40 dist-test-slave-pww0 krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:43:40.127650 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:40.109864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.30.33.3:35511)
0810 23:43:40.110052 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:40.110057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:40.110080 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:43:40.110331 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:40.110336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:40.110403 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:40.110510 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:40.112000 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:40.112885 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:40.114241 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:40.114803 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:40.114931 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:40.120136 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:40.120167 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:40.120193 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:40.120249 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:40.124461 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:40.125262 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:40.125272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:40.125275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:40.125493 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:40.125887 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:40.125893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:40.125896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:40.126414 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:40.126741 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:40.126955 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 23:43:40.127191 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:43:40.128434 1237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:40.257012 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 978
I20260810 23:43:40.286494 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 32746
I20260810 23:43:40.287636 407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:40.399333 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 32746
2026-08-10T23:43:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T23:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:40Z Disabled control of system clock
I20260810 23:43:40.504359 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:45821
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:43:40.860271 1342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:40.860602 1342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:40.860715 1342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:40.868341 1342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:40.868477 1342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:40.868518 1342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:40.868602 1342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:40.868665 1342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:40.878437 1342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:45821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:40.879832 1342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:40.881717 1342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:40.894284 1349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:40.894316 1354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:40.894255 1352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:40.895166 1342 server_base.cc:1034] running on GCE node
I20260810 23:43:40.896152 1342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:40.897729 1342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:40.898955 1342 hybrid_clock.cc:648] HybridClock initialized: now 1786405420898895 us; error 64 us; skew 500 ppm
I20260810 23:43:40.899328 1342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:40.901755 1342 webserver.cc:460] Webserver started at http://127.30.33.62:41327/ using document root <none> and password file <none>
I20260810 23:43:40.902514 1342 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:40.902637 1342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:40.902971 1342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:40.906087 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3"
format_stamp: "Formatted at 2026-08-10 23:43:40 on dist-test-slave-pww0"
server_key: "b0f813923dc8592ee59b2699b3cfbf7b"
server_key_iv: "1f5e97697b686c964c3b24f02938bc18"
server_key_version: "encryptionkey@0"
I20260810 23:43:40.906953 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3"
format_stamp: "Formatted at 2026-08-10 23:43:40 on dist-test-slave-pww0"
server_key: "b0f813923dc8592ee59b2699b3cfbf7b"
server_key_iv: "1f5e97697b686c964c3b24f02938bc18"
server_key_version: "encryptionkey@0"
I20260810 23:43:40.912572 1342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 23:43:40.916507 1362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:40.918098 1342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:40.918289 1342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3"
format_stamp: "Formatted at 2026-08-10 23:43:40 on dist-test-slave-pww0"
server_key: "b0f813923dc8592ee59b2699b3cfbf7b"
server_key_iv: "1f5e97697b686c964c3b24f02938bc18"
server_key_version: "encryptionkey@0"
I20260810 23:43:40.918509 1342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:40.931067 1342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:40.932025 1342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:40.932305 1342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:40.950450 1342 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:45821
I20260810 23:43:40.950491 1460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:45821 every 8 connection(s)
I20260810 23:43:40.952055 1342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:40.956497 1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:40.958014 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 1342
I20260810 23:43:40.958182 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:40.958671 30852 external_mini_cluster.cc:1428] Setting key 9ad239b817e27304cfb10cb399e59551
I20260810 23:43:40.971609 1461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3: Bootstrap starting.
I20260810 23:43:40.974597 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:40.960821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.30.33.62:45821)
0810 23:43:40.962028 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:40.962044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:40.962099 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:43:40.963950 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:40.963989 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:40.964709 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:40.965162 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:40.965766 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:40.967750 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:40.969323 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:40.970437 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:40.970627 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:40.972000 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:40.972012 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:40.972030 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:40.972634 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:40.972727 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:40.972754 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:40.972894 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:40.973186 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 23:43:40.973545 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":392,"threads_started":1}
I20260810 23:43:40.975014 1461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:40.976167 1461 log.cc:824] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:40.979257 1461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3: No bootstrap required, opened a new log
I20260810 23:43:40.986949 1461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45821 } }
I20260810 23:43:40.987411 1461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:40.987540 1461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1a0ed2538d41dd8cc05f5658b4a8a3, State: Initialized, Role: FOLLOWER
I20260810 23:43:40.988222 1461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45821 } }
I20260810 23:43:40.988484 1461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:40.988615 1461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:40.988777 1461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:40.991061 1461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45821 } }
I20260810 23:43:40.991736 1461 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e1a0ed2538d41dd8cc05f5658b4a8a3; no voters:
I20260810 23:43:40.992389 1461 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:40.992502 1471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:40.993718 1471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2e1a0ed2538d41dd8cc05f5658b4a8a3, State: Running, Role: LEADER
I20260810 23:43:40.994240 1471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45821 } }
I20260810 23:43:40.994663 1461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:40.998203 1476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e1a0ed2538d41dd8cc05f5658b4a8a3. Latest consensus state: current_term: 1 leader_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45821 } } }
I20260810 23:43:40.998704 1476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:40.998205 1473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1a0ed2538d41dd8cc05f5658b4a8a3" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 45821 } } }
I20260810 23:43:40.998962 1473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:41.000216 1485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:41.006052 1485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:41.016899 1485 catalog_manager.cc:1347] Generated new cluster ID: 37929e05645b4d829377ec794dd910cc
I20260810 23:43:41.017035 1485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:41.035998 1485 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:41.037364 1485 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:41.046450 1485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3: Generated new TSK 0
I20260810 23:43:41.047384 1485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:43:41.062014 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45821
--builtin_ntp_servers=127.30.33.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:43:41.456025 1502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:41.456396 1502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:41.456504 1502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:41.464030 1502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:41.464154 1502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:41.464274 1502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:43:41.473783 1502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:41.475118 1502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:41.477085 1502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:41.490525 1507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:41.490450 1513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:41.490489 1509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:41.491806 1502 server_base.cc:1034] running on GCE node
I20260810 23:43:41.492506 1502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:41.493559 1502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:41.494786 1502 hybrid_clock.cc:648] HybridClock initialized: now 1786405421494720 us; error 49 us; skew 500 ppm
I20260810 23:43:41.495147 1502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:41.497700 1502 webserver.cc:460] Webserver started at http://127.30.33.1:42095/ using document root <none> and password file <none>
I20260810 23:43:41.498435 1502 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:41.498533 1502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:41.498816 1502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:41.502194 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "8074c3609c1b42ecb8b094b85d9b6ca6"
format_stamp: "Formatted at 2026-08-10 23:43:41 on dist-test-slave-pww0"
server_key: "a0fb72c5ff9654b992834e41f593fc36"
server_key_iv: "6d5fe50cdc509d9d90ce2ac360d8a7b0"
server_key_version: "encryptionkey@0"
I20260810 23:43:41.503070 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "8074c3609c1b42ecb8b094b85d9b6ca6"
format_stamp: "Formatted at 2026-08-10 23:43:41 on dist-test-slave-pww0"
server_key: "a0fb72c5ff9654b992834e41f593fc36"
server_key_iv: "6d5fe50cdc509d9d90ce2ac360d8a7b0"
server_key_version: "encryptionkey@0"
I20260810 23:43:41.509079 1502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 23:43:41.513123 1523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:41.514681 1502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:41.514840 1502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "8074c3609c1b42ecb8b094b85d9b6ca6"
format_stamp: "Formatted at 2026-08-10 23:43:41 on dist-test-slave-pww0"
server_key: "a0fb72c5ff9654b992834e41f593fc36"
server_key_iv: "6d5fe50cdc509d9d90ce2ac360d8a7b0"
server_key_version: "encryptionkey@0"
I20260810 23:43:41.515046 1502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:41.535995 1502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:41.536926 1502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:41.537220 1502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:41.538275 1502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:41.539850 1502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:41.539940 1502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:41.540025 1502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:41.540089 1502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:41.574910 1502 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:38119
I20260810 23:43:41.574995 1737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:38119 every 8 connection(s)
I20260810 23:43:41.576356 1502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:43:41.583953 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 1502
I20260810 23:43:41.584165 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:43:41.584594 30852 external_mini_cluster.cc:1428] Setting key 8ad158efd5bc7e93b8a9646bdfb9d61c
I20260810 23:43:41.587811 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45821
--builtin_ntp_servers=127.30.33.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:43:41.589888 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:41.579027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:43015 (local address 127.30.33.62:45821)
0810 23:43:41.579530 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:41.579536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:41.580837 (+ 1301us) server_negotiation.cc:407] Connection header received
0810 23:43:41.582636 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:41.582640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:41.582714 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:41.582817 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:41.584125 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:41.584903 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:41.586824 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:41.587233 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:41.587343 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:41.589021 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:41.589027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:41.589030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:41.589051 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:41.589108 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:41.589124 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:41.589207 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:41.589508 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 23:43:41.589603 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260810 23:43:41.591291 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:41.579700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45821 (local address 127.30.33.1:43015)
0810 23:43:41.580669 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:41.580714 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:41.582371 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:41.583011 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:41.583019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:41.583108 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:41.583887 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:41.583915 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:41.585078 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:41.585083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:41.586661 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:41.586675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:41.587498 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:41.587507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:41.587824 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:41.588608 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:41.588611 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:41.588669 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:41.588787 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:41.589222 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:41.589234 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:41.589251 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:43:41.589442 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 23:43:41.589893 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":285,"threads_started":1}
I20260810 23:43:41.592928 1742 heartbeater.cc:347] Connected to a master server at 127.30.33.62:45821
I20260810 23:43:41.593288 1742 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:41.594247 1742 heartbeater.cc:511] Master 127.30.33.62:45821 requested a full tablet report, sending...
I20260810 23:43:41.596598 1392 ts_manager.cc:194] Registered new tserver with Master: 8074c3609c1b42ecb8b094b85d9b6ca6 (127.30.33.1:38119)
I20260810 23:43:41.598186 1392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.1:43015
W20260810 23:43:41.948987 1746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:41.949314 1746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:41.949436 1746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:41.957249 1746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:41.957373 1746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:41.957525 1746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:43:41.967192 1746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:41.968603 1746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:41.970511 1746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:41.984437 1751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:41.984506 1755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:41.984539 1752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:41.985814 1746 server_base.cc:1034] running on GCE node
I20260810 23:43:41.986604 1746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:41.987955 1746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:41.989209 1746 hybrid_clock.cc:648] HybridClock initialized: now 1786405421989145 us; error 51 us; skew 500 ppm
I20260810 23:43:41.989579 1746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:41.992208 1746 webserver.cc:460] Webserver started at http://127.30.33.2:33331/ using document root <none> and password file <none>
I20260810 23:43:41.992995 1746 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:41.993111 1746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:41.993448 1746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:41.996763 1746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "6511d51d365241edb24a7bf8b694e256"
format_stamp: "Formatted at 2026-08-10 23:43:41 on dist-test-slave-pww0"
server_key: "df60325896fff54ee138984b64b3c7c0"
server_key_iv: "6ba8230a14be6521ed12503dd5b94ba2"
server_key_version: "encryptionkey@0"
I20260810 23:43:41.997634 1746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "6511d51d365241edb24a7bf8b694e256"
format_stamp: "Formatted at 2026-08-10 23:43:41 on dist-test-slave-pww0"
server_key: "df60325896fff54ee138984b64b3c7c0"
server_key_iv: "6ba8230a14be6521ed12503dd5b94ba2"
server_key_version: "encryptionkey@0"
I20260810 23:43:42.003434 1746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:43:42.007681 1766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:42.009287 1746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:43:42.009500 1746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "6511d51d365241edb24a7bf8b694e256"
format_stamp: "Formatted at 2026-08-10 23:43:41 on dist-test-slave-pww0"
server_key: "df60325896fff54ee138984b64b3c7c0"
server_key_iv: "6ba8230a14be6521ed12503dd5b94ba2"
server_key_version: "encryptionkey@0"
I20260810 23:43:42.009740 1746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:42.031970 1746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:42.032922 1746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:42.033212 1746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:42.034338 1746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:42.036087 1746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:42.036191 1746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:42.036325 1746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:42.036402 1746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:42.070518 1746 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:46415
I20260810 23:43:42.070595 1969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:46415 every 8 connection(s)
I20260810 23:43:42.071980 1746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:43:42.073015 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 1746
I20260810 23:43:42.073246 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:43:42.073887 30852 external_mini_cluster.cc:1428] Setting key f54a1872bcd5df64cb12b2614e99edea
I20260810 23:43:42.077355 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45821
--builtin_ntp_servers=127.30.33.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:43:42.086378 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.074963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:60983 (local address 127.30.33.62:45821)
0810 23:43:42.075133 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.075138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.076997 (+ 1859us) server_negotiation.cc:407] Connection header received
0810 23:43:42.079139 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.079145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.079308 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.079428 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:42.080746 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.081484 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.083349 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.083864 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.083989 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.085508 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:42.085513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:42.085516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:42.085538 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:42.085574 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:42.085587 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:42.085665 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.085907 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.086034 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:43:42.087555 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.075810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45821 (local address 127.30.33.2:60983)
0810 23:43:42.076843 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:42.076884 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:42.078777 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:42.079765 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:42.079774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:42.079898 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:42.080549 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.080568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.081661 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.081666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.083158 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.083179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.084114 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.084122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.084370 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.085117 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:42.085119 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:42.085167 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:42.085272 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:42.085690 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:42.085701 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:42.085709 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:43:42.085883 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 23:43:42.086363 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":342,"threads_started":1}
I20260810 23:43:42.089241 1970 heartbeater.cc:347] Connected to a master server at 127.30.33.62:45821
I20260810 23:43:42.089584 1970 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:42.090677 1970 heartbeater.cc:511] Master 127.30.33.62:45821 requested a full tablet report, sending...
I20260810 23:43:42.092382 1392 ts_manager.cc:194] Registered new tserver with Master: 6511d51d365241edb24a7bf8b694e256 (127.30.33.2:46415)
I20260810 23:43:42.093474 1392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.2:60983
W20260810 23:43:42.433605 1973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:42.433928 1973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:42.434015 1973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:42.441454 1973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:42.441583 1973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:42.441741 1973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:43:42.451359 1973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:42.452839 1973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:42.454797 1973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:42.468348 1989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:42.468528 1992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:42.468348 1986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:42.469165 1973 server_base.cc:1034] running on GCE node
I20260810 23:43:42.469882 1973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:42.470975 1973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:42.472193 1973 hybrid_clock.cc:648] HybridClock initialized: now 1786405422472140 us; error 42 us; skew 500 ppm
I20260810 23:43:42.472543 1973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:42.474962 1973 webserver.cc:460] Webserver started at http://127.30.33.3:44581/ using document root <none> and password file <none>
I20260810 23:43:42.475769 1973 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:42.475878 1973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:42.476203 1973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:42.479328 1973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "83af357dc40248b7bfc92d3f3b211c2c"
format_stamp: "Formatted at 2026-08-10 23:43:42 on dist-test-slave-pww0"
server_key: "f7473eb7aa457b0459c79d1885e8c0c1"
server_key_iv: "c2a40280130f5ebc6e725dbeb0a18f6f"
server_key_version: "encryptionkey@0"
I20260810 23:43:42.480269 1973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "83af357dc40248b7bfc92d3f3b211c2c"
format_stamp: "Formatted at 2026-08-10 23:43:42 on dist-test-slave-pww0"
server_key: "f7473eb7aa457b0459c79d1885e8c0c1"
server_key_iv: "c2a40280130f5ebc6e725dbeb0a18f6f"
server_key_version: "encryptionkey@0"
I20260810 23:43:42.485955 1973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:43:42.490219 2003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:42.491802 1973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:43:42.491973 1973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "83af357dc40248b7bfc92d3f3b211c2c"
format_stamp: "Formatted at 2026-08-10 23:43:42 on dist-test-slave-pww0"
server_key: "f7473eb7aa457b0459c79d1885e8c0c1"
server_key_iv: "c2a40280130f5ebc6e725dbeb0a18f6f"
server_key_version: "encryptionkey@0"
I20260810 23:43:42.492197 1973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:42.514376 1973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:42.515347 1973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:42.515674 1973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:42.516755 1973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:42.518414 1973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:42.518517 1973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:42.518635 1973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:42.518707 1973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:42.551445 1973 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:43505
I20260810 23:43:42.551586 2223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:43505 every 8 connection(s)
I20260810 23:43:42.552850 1973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:43:42.554097 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 1973
I20260810 23:43:42.554301 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:43:42.554886 30852 external_mini_cluster.cc:1428] Setting key dd6d149d806f512e73edb732afc2eaeb
I20260810 23:43:42.566179 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.555841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:41985 (local address 127.30.33.62:45821)
0810 23:43:42.556069 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.556075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.557792 (+ 1717us) server_negotiation.cc:407] Connection header received
0810 23:43:42.559643 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.559649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.559750 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.559840 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:42.560965 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.561705 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.563313 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.563770 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.563890 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.565299 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:42.565304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:42.565307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:42.565329 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:42.565364 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:42.565377 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:42.565455 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.565677 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.565799 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:43:42.567409 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.556557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:45821 (local address 127.30.33.3:41985)
0810 23:43:42.557626 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:42.557674 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:42.559341 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:42.560093 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:42.560102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:42.560199 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:42.560802 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.560819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.561865 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.561869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.563156 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.563169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.564008 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.564015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.564265 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.564956 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:42.564958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:42.565003 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:42.565097 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:42.565467 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:42.565479 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:42.565487 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:43:42.565655 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 23:43:42.566094 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":381,"threads_started":1}
I20260810 23:43:42.568948 2224 heartbeater.cc:347] Connected to a master server at 127.30.33.62:45821
I20260810 23:43:42.569305 2224 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:42.570218 2224 heartbeater.cc:511] Master 127.30.33.62:45821 requested a full tablet report, sending...
I20260810 23:43:42.571746 1392 ts_manager.cc:194] Registered new tserver with Master: 83af357dc40248b7bfc92d3f3b211c2c (127.30.33.3:43505)
I20260810 23:43:42.572547 1392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.3:41985
I20260810 23:43:42.583433 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:43:42.593784 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.587686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.30.33.62:45821)
0810 23:43:42.587950 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.587957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.588410 (+ 453us) server_negotiation.cc:407] Connection header received
0810 23:43:42.588564 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.588568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.588639 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.588727 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:42.589219 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.589931 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.591397 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.591861 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.591955 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.593072 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:42.593076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:42.593080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:42.593099 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:42.593143 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:42.593156 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:42.593279 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.593397 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.593453 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:43:42.601457 1742 heartbeater.cc:503] Master 127.30.33.62:45821 was elected leader, sending a full tablet report...
I20260810 23:43:42.597963 1392 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:43:42.605412 1392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:43:42.631697 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.617825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:38119 (local address 127.0.0.1:47884)
0810 23:43:42.618932 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:42.618989 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:42.619170 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:42.624249 (+ 5079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:42.624254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:42.624289 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:42.624846 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.624861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.626736 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.626743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.629118 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.629135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.630860 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.630866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.631061 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.631098 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:43:42.631160 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:43:42.631309 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":74,"thread_start_us":4714,"threads_started":1}
I20260810 23:43:42.632175 2259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.618150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43505 (local address 127.0.0.1:58352)
0810 23:43:42.622541 (+ 4391us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:42.622565 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:42.622818 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:42.623939 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:42.623947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:42.624000 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:42.624563 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.624578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.626243 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.626249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.629127 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.629137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.631347 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.631353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.631566 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.631600 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:43:42.631703 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:43:42.631827 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4251,"mutex_wait_us":270,"spinlock_wait_cycles":15488,"thread_start_us":2336,"threads_started":1}
I20260810 23:43:42.632284 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.619242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58352 (local address 127.30.33.3:43505)
0810 23:43:42.619886 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.619895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.622685 (+ 2790us) server_negotiation.cc:407] Connection header received
0810 23:43:42.623024 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.623034 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.623364 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.623573 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:42.624824 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.626073 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.629392 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.630776 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.630910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.631202 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.631776 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.631960 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":889,"threads_started":1}
I20260810 23:43:42.632747 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.619974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.30.33.2:46415)
0810 23:43:42.620486 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.620495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.622904 (+ 2409us) server_negotiation.cc:407] Connection header received
0810 23:43:42.623176 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.623184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.623384 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.623609 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:42.624831 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.626362 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.628757 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.629961 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.630093 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.630375 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.631444 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.631674 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":375,"threads_started":1}
I20260810 23:43:42.633060 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.618047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.30.33.1:38119)
0810 23:43:42.623559 (+ 5512us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.623568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.623600 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:43:42.623714 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.623721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.623982 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.624236 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:42.625044 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.626565 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.629376 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.630636 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.632278 (+ 1642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.632560 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.632619 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.632749 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5292,"thread_start_us":5890,"threads_started":1}
I20260810 23:43:42.633244 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.618814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:46415 (local address 127.0.0.1:34930)
0810 23:43:42.622790 (+ 3976us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:42.622805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:42.623005 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:42.623939 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:42.623947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:42.623992 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:42.624618 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.624636 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.626543 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.626548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.628554 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.628568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.630519 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.630526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.630754 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.630818 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 23:43:42.630957 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:43:42.631177 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3867,"mutex_wait_us":23,"spinlock_wait_cycles":1280,"thread_start_us":3427,"threads_started":1}
I20260810 23:43:42.636411 2096 tablet_service.cc:1467] Processing CreateTablet for tablet 5a43c224ab684f84a45c6b8542c90419 (DEFAULT_TABLE table=test-table [id=40401eb2808a4ba58e79cd40715a404d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:42.636823 1625 tablet_service.cc:1467] Processing CreateTablet for tablet 5a43c224ab684f84a45c6b8542c90419 (DEFAULT_TABLE table=test-table [id=40401eb2808a4ba58e79cd40715a404d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:42.636823 1852 tablet_service.cc:1467] Processing CreateTablet for tablet 5a43c224ab684f84a45c6b8542c90419 (DEFAULT_TABLE table=test-table [id=40401eb2808a4ba58e79cd40715a404d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:42.637820 2096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a43c224ab684f84a45c6b8542c90419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:42.638262 1852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a43c224ab684f84a45c6b8542c90419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:42.638252 1625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a43c224ab684f84a45c6b8542c90419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:42.651165 2262 tablet_bootstrap.cc:492] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Bootstrap starting.
I20260810 23:43:42.653719 2265 tablet_bootstrap.cc:492] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Bootstrap starting.
I20260810 23:43:42.655512 2267 tablet_bootstrap.cc:492] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Bootstrap starting.
I20260810 23:43:42.657194 2262 tablet_bootstrap.cc:654] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:42.657667 2265 tablet_bootstrap.cc:654] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:42.658654 2262 log.cc:824] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:42.658705 2267 tablet_bootstrap.cc:654] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:42.658885 2265 log.cc:824] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:42.659911 2267 log.cc:824] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:42.661906 2262 tablet_bootstrap.cc:492] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: No bootstrap required, opened a new log
I20260810 23:43:42.662196 2262 ts_tablet_manager.cc:1397] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 23:43:42.662947 2267 tablet_bootstrap.cc:492] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: No bootstrap required, opened a new log
I20260810 23:43:42.662947 2265 tablet_bootstrap.cc:492] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: No bootstrap required, opened a new log
I20260810 23:43:42.663228 2267 ts_tablet_manager.cc:1397] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 23:43:42.663240 2265 ts_tablet_manager.cc:1397] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 23:43:42.670064 2262 raft_consensus.cc:348] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.670507 2262 raft_consensus.cc:374] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:42.670610 2262 raft_consensus.cc:729] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83af357dc40248b7bfc92d3f3b211c2c, State: Initialized, Role: FOLLOWER
I20260810 23:43:42.671255 2262 consensus_queue.cc:260] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.672447 2224 heartbeater.cc:503] Master 127.30.33.62:45821 was elected leader, sending a full tablet report...
I20260810 23:43:42.673194 2262 ts_tablet_manager.cc:1428] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 23:43:42.673996 2267 raft_consensus.cc:348] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.673995 2265 raft_consensus.cc:348] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.674494 2265 raft_consensus.cc:374] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:42.674615 2265 raft_consensus.cc:729] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6511d51d365241edb24a7bf8b694e256, State: Initialized, Role: FOLLOWER
I20260810 23:43:42.674824 2267 raft_consensus.cc:374] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:42.675098 2267 raft_consensus.cc:729] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8074c3609c1b42ecb8b094b85d9b6ca6, State: Initialized, Role: FOLLOWER
I20260810 23:43:42.675347 2265 consensus_queue.cc:260] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.675792 2267 consensus_queue.cc:260] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.676653 1970 heartbeater.cc:503] Master 127.30.33.62:45821 was elected leader, sending a full tablet report...
I20260810 23:43:42.677479 2265 ts_tablet_manager.cc:1428] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 23:43:42.679431 2267 ts_tablet_manager.cc:1428] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260810 23:43:42.804788 2229 tablet.cc:2369] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:43:42.824232 1971 tablet.cc:2369] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:43:42.828804 1743 tablet.cc:2369] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:42.895010 2274 raft_consensus.cc:482] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:43:42.895354 2274 raft_consensus.cc:504] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.897269 2274 leader_election.cc:288] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8074c3609c1b42ecb8b094b85d9b6ca6 (127.30.33.1:38119), 83af357dc40248b7bfc92d3f3b211c2c (127.30.33.3:43505)
I20260810 23:43:42.906948 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.897829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:53035 (local address 127.30.33.3:43505)
0810 23:43:42.898003 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.898009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.898728 (+ 719us) server_negotiation.cc:407] Connection header received
0810 23:43:42.898954 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.898960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.899044 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.899142 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:42.899980 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.901090 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.904340 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.905690 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.905844 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.905969 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.906487 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.906662 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:43:42.906944 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.897721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:52897 (local address 127.30.33.1:38119)
0810 23:43:42.897934 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.897941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.898761 (+ 820us) server_negotiation.cc:407] Connection header received
0810 23:43:42.898954 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.898960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.899045 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.899142 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:42.899963 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.901054 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.904340 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.905686 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.905844 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.905967 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.906475 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.906663 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:43:42.907007 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.897707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:38119 (local address 127.30.33.2:52897)
0810 23:43:42.898652 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:42.898665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:42.898813 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:42.899178 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:42.899183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:42.899229 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:42.899792 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.899808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.901210 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.901216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.904040 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.904061 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.905959 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.905966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.906223 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.906288 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 23:43:42.906395 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:43:42.906562 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":235,"spinlock_wait_cycles":67456,"thread_start_us":412,"threads_started":1}
I20260810 23:43:42.907032 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.897680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43505 (local address 127.30.33.2:53035)
0810 23:43:42.898588 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:42.898609 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:42.898814 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:42.899178 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:42.899183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:42.899225 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:42.899800 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.899809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.901217 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.901221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.904040 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:42.904061 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.905958 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.905966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:42.906223 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.906313 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 23:43:42.906395 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:43:42.906550 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":173,"spinlock_wait_cycles":34688,"thread_start_us":534,"threads_started":1}
I20260810 23:43:42.908490 2135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a43c224ab684f84a45c6b8542c90419" candidate_uuid: "6511d51d365241edb24a7bf8b694e256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83af357dc40248b7bfc92d3f3b211c2c" is_pre_election: true
I20260810 23:43:42.908489 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a43c224ab684f84a45c6b8542c90419" candidate_uuid: "6511d51d365241edb24a7bf8b694e256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" is_pre_election: true
I20260810 23:43:42.909152 2135 raft_consensus.cc:2454] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6511d51d365241edb24a7bf8b694e256 in term 0.
I20260810 23:43:42.909152 1662 raft_consensus.cc:2454] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6511d51d365241edb24a7bf8b694e256 in term 0.
I20260810 23:43:42.909988 1770 leader_election.cc:302] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6511d51d365241edb24a7bf8b694e256, 83af357dc40248b7bfc92d3f3b211c2c; no voters:
I20260810 23:43:42.910441 2274 raft_consensus.cc:2781] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:43:42.910605 2274 raft_consensus.cc:482] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:43:42.910712 2274 raft_consensus.cc:3037] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:42.913000 2274 raft_consensus.cc:504] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.913641 2274 leader_election.cc:288] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [CANDIDATE]: Term 1 election: Requested vote from peers 8074c3609c1b42ecb8b094b85d9b6ca6 (127.30.33.1:38119), 83af357dc40248b7bfc92d3f3b211c2c (127.30.33.3:43505)
I20260810 23:43:42.914142 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a43c224ab684f84a45c6b8542c90419" candidate_uuid: "6511d51d365241edb24a7bf8b694e256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6"
I20260810 23:43:42.914196 2135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a43c224ab684f84a45c6b8542c90419" candidate_uuid: "6511d51d365241edb24a7bf8b694e256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83af357dc40248b7bfc92d3f3b211c2c"
I20260810 23:43:42.914361 1662 raft_consensus.cc:3037] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:42.914402 2135 raft_consensus.cc:3037] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:42.916700 2135 raft_consensus.cc:2454] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6511d51d365241edb24a7bf8b694e256 in term 1.
I20260810 23:43:42.916772 1662 raft_consensus.cc:2454] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6511d51d365241edb24a7bf8b694e256 in term 1.
I20260810 23:43:42.917172 1770 leader_election.cc:302] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6511d51d365241edb24a7bf8b694e256, 8074c3609c1b42ecb8b094b85d9b6ca6, 83af357dc40248b7bfc92d3f3b211c2c; no voters:
I20260810 23:43:42.917516 2274 raft_consensus.cc:2781] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:42.917984 2274 raft_consensus.cc:686] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 1 LEADER]: Becoming Leader. State: Replica: 6511d51d365241edb24a7bf8b694e256, State: Running, Role: LEADER
I20260810 23:43:42.918485 2274 consensus_queue.cc:237] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } }
I20260810 23:43:42.923189 1392 catalog_manager.cc:5495] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6511d51d365241edb24a7bf8b694e256 (127.30.33.2). New cstate: current_term: 1 leader_uuid: "6511d51d365241edb24a7bf8b694e256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6511d51d365241edb24a7bf8b694e256" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 46415 } health_report { overall_health: HEALTHY } } }
I20260810 23:43:42.946174 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.940691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.30.33.2:46415)
0810 23:43:42.940886 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.940890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:42.940983 (+ 93us) server_negotiation.cc:407] Connection header received
0810 23:43:42.941227 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:42.941232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:42.941331 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:42.941475 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:43:42.941871 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.942592 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.944046 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:42.944492 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:42.944598 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:42.945090 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:43:42.945615 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:43:42.945774 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:42.945907 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:43:42.945949 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:43:42.955070 1662 raft_consensus.cc:1261] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 1 FOLLOWER]: Refusing update from remote peer 6511d51d365241edb24a7bf8b694e256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:42.955070 2135 raft_consensus.cc:1261] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 1 FOLLOWER]: Refusing update from remote peer 6511d51d365241edb24a7bf8b694e256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:42.956286 2287 consensus_queue.cc:1035] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8074c3609c1b42ecb8b094b85d9b6ca6" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:42.956655 2274 consensus_queue.cc:1035] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83af357dc40248b7bfc92d3f3b211c2c" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:42.972208 2295 mvcc.cc:205] Tried to move back new op lower bound from 7317116612412354560 to 7317116612278214656. Current Snapshot: MvccSnapshot[applied={T|T < 7317116612412354560}]
I20260810 23:43:42.972873 2296 mvcc.cc:205] Tried to move back new op lower bound from 7317116612412354560 to 7317116612278214656. Current Snapshot: MvccSnapshot[applied={T|T < 7317116612412354560}]
I20260810 23:43:42.987193 1391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:42.987571 1391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:42.991951 1391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e1a0ed2538d41dd8cc05f5658b4a8a3: Sending DeleteTablet for 3 replicas of tablet 5a43c224ab684f84a45c6b8542c90419
I20260810 23:43:42.993230 2096 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a43c224ab684f84a45c6b8542c90419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:42 UTC) from {username='slave'} at 127.0.0.1:58352
I20260810 23:43:42.993237 1852 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a43c224ab684f84a45c6b8542c90419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:42 UTC) from {username='slave'} at 127.0.0.1:34930
I20260810 23:43:42.993247 1625 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a43c224ab684f84a45c6b8542c90419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:42 UTC) from {username='slave'} at 127.0.0.1:47884
I20260810 23:43:42.997483 2319 tablet_replica.cc:330] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: stopping tablet replica
I20260810 23:43:42.997848 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1502
I20260810 23:43:42.998000 2319 raft_consensus.cc:2229] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:42.998611 2319 raft_consensus.cc:2258] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:43.008867 2319 ts_tablet_manager.cc:1905] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:43.010568 2316 tablet_replica.cc:330] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: stopping tablet replica
I20260810 23:43:43.010684 2317 tablet_replica.cc:330] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: stopping tablet replica
I20260810 23:43:43.011075 2316 raft_consensus.cc:2229] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:43:43.011169 2317 raft_consensus.cc:2229] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:43.011792 2317 raft_consensus.cc:2258] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:43.011874 2316 raft_consensus.cc:2258] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:43.014254 2317 ts_tablet_manager.cc:1905] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:43.014307 2316 ts_tablet_manager.cc:1905] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:43.017653 2319 ts_tablet_manager.cc:1918] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:43.017868 2319 log.cc:1196] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/5a43c224ab684f84a45c6b8542c90419
I20260810 23:43:43.018453 2319 ts_tablet_manager.cc:1939] T 5a43c224ab684f84a45c6b8542c90419 P 83af357dc40248b7bfc92d3f3b211c2c: Deleting consensus metadata
I20260810 23:43:43.020716 1365 catalog_manager.cc:4841] TS 83af357dc40248b7bfc92d3f3b211c2c (127.30.33.3:43505): tablet 5a43c224ab684f84a45c6b8542c90419 (table test-table [id=40401eb2808a4ba58e79cd40715a404d]) successfully deleted
I20260810 23:43:43.021828 2316 ts_tablet_manager.cc:1918] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:43.022025 2316 log.cc:1196] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/5a43c224ab684f84a45c6b8542c90419
I20260810 23:43:43.022636 2316 ts_tablet_manager.cc:1939] T 5a43c224ab684f84a45c6b8542c90419 P 6511d51d365241edb24a7bf8b694e256: Deleting consensus metadata
I20260810 23:43:43.024672 1367 catalog_manager.cc:4841] TS 6511d51d365241edb24a7bf8b694e256 (127.30.33.2:46415): tablet 5a43c224ab684f84a45c6b8542c90419 (table test-table [id=40401eb2808a4ba58e79cd40715a404d]) successfully deleted
I20260810 23:43:43.025098 2317 ts_tablet_manager.cc:1918] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:43.025280 2317 log.cc:1196] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/5a43c224ab684f84a45c6b8542c90419
I20260810 23:43:43.025751 2317 ts_tablet_manager.cc:1939] T 5a43c224ab684f84a45c6b8542c90419 P 8074c3609c1b42ecb8b094b85d9b6ca6: Deleting consensus metadata
I20260810 23:43:43.027848 1363 catalog_manager.cc:4841] TS 8074c3609c1b42ecb8b094b85d9b6ca6 (127.30.33.1:38119): tablet 5a43c224ab684f84a45c6b8542c90419 (table test-table [id=40401eb2808a4ba58e79cd40715a404d]) successfully deleted
I20260810 23:43:43.029382 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:42.999397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.30.33.1:38119)
0810 23:43:42.999625 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:42.999633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:43.021593 (+ 21960us) server_negotiation.cc:407] Connection header received
0810 23:43:43.021814 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:43.021820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:43.021949 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:43.022071 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:43.022734 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:43.023994 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:43.025995 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:43.026626 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:43.026782 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:43.028034 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:43.028045 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:43.028064 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:43.028646 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:43.028701 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:43.028731 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:43.028819 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:43.028987 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:43:43.029081 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 23:43:43.030082 1729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:43.142493 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 1502
I20260810 23:43:43.174383 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1746
I20260810 23:43:43.186209 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:43.175203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.30.33.2:46415)
0810 23:43:43.175445 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:43.175451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:43.175497 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:43:43.175628 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:43.175634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:43.175758 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:43.175856 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:43.176333 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:43.177216 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:43.178677 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:43.179150 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:43.179271 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:43.184586 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:43.184597 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:43.184614 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:43.185332 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:43.185393 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:43.185423 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:43.185541 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:43.185733 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 23:43:43.185865 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 23:43:43.187009 1961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:43.303678 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 1746
I20260810 23:43:43.329545 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1973
I20260810 23:43:43.336258 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:43.330316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.30.33.3:43505)
0810 23:43:43.330474 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:43.330478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:43.330595 (+ 117us) server_negotiation.cc:407] Connection header received
0810 23:43:43.330716 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:43.330727 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:43.330831 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:43.330917 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:43.331323 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:43.332105 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:43.333417 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:43.333822 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:43.333921 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:43.334920 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:43.334931 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:43.334946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:43.335594 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:43.335650 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:43.335679 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:43.335780 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:43.335881 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:43:43.335981 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:43:43.336949 2214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:43.425326 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 1973
I20260810 23:43:43.448630 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 1342
I20260810 23:43:43.449824 1450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:43.561852 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 1342
2026-08-10T23:43:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:43:43 dist-test-slave-pww0 krb5kdc[2337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:43:43 dist-test-slave-pww0 krb5kdc[2337](info): set up 2 sockets
Aug 10 23:43:43 dist-test-slave-pww0 krb5kdc[2337](info): commencing operation
krb5kdc: starting...
W20260810 23:43:44.877445 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.006s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:43:45 dist-test-slave-pww0 krb5kdc[2337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:43:45.225044 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:33925
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:33925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:45.582000 2356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:45.582340 2356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:45.582433 2356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:45.590610 2356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:45.590749 2356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:45.590821 2356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:45.590907 2356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:45.590955 2356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:45.601444 2356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:33925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:33925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:45.602871 2356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:45.604267 2356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:45.618299 2367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:45.618208 2365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:45.618343 2362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:45.619201 2356 server_base.cc:1034] running on GCE node
I20260810 23:43:45.620105 2356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:45.621654 2356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:45.622900 2356 hybrid_clock.cc:648] HybridClock initialized: now 1786405425622813 us; error 83 us; skew 500 ppm
I20260810 23:43:45.623277 2356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:45 dist-test-slave-pww0 krb5kdc[2337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405425, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:45.626565 2356 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:43:45.628269 2356 webserver.cc:460] Webserver started at http://127.30.33.62:40847/ using document root <none> and password file <none>
I20260810 23:43:45.629010 2356 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:45.629127 2356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:45.629452 2356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:45.632757 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "e6ea16aa8c684dde8a854fbd06d729e9"
format_stamp: "Formatted at 2026-08-10 23:43:45 on dist-test-slave-pww0"
server_key: "8497e3af484ee04ced442e0f3edb6310"
server_key_iv: "1a29d9fbf5d365cf71d816c7e7cc03fa"
server_key_version: "encryptionkey@0"
I20260810 23:43:45.633579 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "e6ea16aa8c684dde8a854fbd06d729e9"
format_stamp: "Formatted at 2026-08-10 23:43:45 on dist-test-slave-pww0"
server_key: "8497e3af484ee04ced442e0f3edb6310"
server_key_iv: "1a29d9fbf5d365cf71d816c7e7cc03fa"
server_key_version: "encryptionkey@0"
I20260810 23:43:45.639207 2356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:43:45.642982 2384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:45.644465 2356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:45.644652 2356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "e6ea16aa8c684dde8a854fbd06d729e9"
format_stamp: "Formatted at 2026-08-10 23:43:45 on dist-test-slave-pww0"
server_key: "8497e3af484ee04ced442e0f3edb6310"
server_key_iv: "1a29d9fbf5d365cf71d816c7e7cc03fa"
server_key_version: "encryptionkey@0"
I20260810 23:43:45.644866 2356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:45.671993 2356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:45.676250 2356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:45.676533 2356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:45.695286 2356 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:33925
I20260810 23:43:45.695318 2490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:33925 every 8 connection(s)
I20260810 23:43:45.696854 2356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:45.699672 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 2356
I20260810 23:43:45.699860 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:45.700297 30852 external_mini_cluster.cc:1428] Setting key aebdc9856264ca66c76e042514f1493a
I20260810 23:43:45.700966 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:43:45 dist-test-slave-pww0 krb5kdc[2337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:45.716854 2491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9: Bootstrap starting.
I20260810 23:43:45.720865 2491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:45.722342 2491 log.cc:824] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:45.725456 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:45.702110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.30.33.62:33925)
0810 23:43:45.702877 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:45.702889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:45.702937 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:43:45.704580 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:45.704603 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:45.705025 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:45.705380 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:45.707536 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:45.708939 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:45.710906 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:45.711649 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:45.711807 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:45.717349 (+ 5542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:45.717358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:45.717375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:45.717425 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:45.720704 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:45.721735 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:45.721744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:45.721751 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:45.721943 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:45.722458 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:45.722462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:45.722464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:45.723107 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:45.723431 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:45.723981 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 23:43:45.724453 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":344,"threads_started":1}
I20260810 23:43:45.725795 2491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9: No bootstrap required, opened a new log
I20260810 23:43:45.734159 2491 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33925 } }
I20260810 23:43:45.734645 2491 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:45.734773 2491 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ea16aa8c684dde8a854fbd06d729e9, State: Initialized, Role: FOLLOWER
I20260810 23:43:45.735404 2491 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33925 } }
I20260810 23:43:45.735793 2491 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:45.735944 2491 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:45.736130 2491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:45.738572 2491 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33925 } }
I20260810 23:43:45.739142 2491 leader_election.cc:302] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6ea16aa8c684dde8a854fbd06d729e9; no voters:
I20260810 23:43:45.739822 2491 leader_election.cc:288] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:45.739954 2499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:45.740660 2499 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [term 1 LEADER]: Becoming Leader. State: Replica: e6ea16aa8c684dde8a854fbd06d729e9, State: Running, Role: LEADER
I20260810 23:43:45.741250 2499 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33925 } }
I20260810 23:43:45.742211 2491 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:45.744402 2502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33925 } } }
I20260810 23:43:45.744706 2503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6ea16aa8c684dde8a854fbd06d729e9. Latest consensus state: current_term: 1 leader_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ea16aa8c684dde8a854fbd06d729e9" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 33925 } } }
I20260810 23:43:45.744985 2502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:45.745039 2503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:45.747359 2514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:45.752861 2514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:45.762395 2514 catalog_manager.cc:1347] Generated new cluster ID: 53e78ce82d014353ba41aff0a2e094c0
I20260810 23:43:45.762534 2514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:45.775326 2514 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:45.777411 2514 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:45.789834 2514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9: Generated new TSK 0
I20260810 23:43:45.791110 2514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
Entry for principal kudu/127.30.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab.
I20260810 23:43:45.882397 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33925
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:46.252820 2533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:46.253153 2533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:46.253298 2533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:46.260962 2533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:46.261112 2533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:46.261274 2533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:43:46.271665 2533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.1.keytab
--principal=kudu/127.30.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:46.273070 2533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:46.274458 2533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:46.288026 2542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:46.287995 2541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:46.288056 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:46.289600 2533 server_base.cc:1034] running on GCE node
I20260810 23:43:46.290313 2533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:46.291349 2533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:46.292588 2533 hybrid_clock.cc:648] HybridClock initialized: now 1786405426292510 us; error 55 us; skew 500 ppm
I20260810 23:43:46.292960 2533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:46 dist-test-slave-pww0 krb5kdc[2337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405426, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:46.296308 2533 init.cc:377] Logged in from keytab as kudu/127.30.33.1@KRBTEST.COM (short username kudu)
I20260810 23:43:46.297837 2533 webserver.cc:460] Webserver started at http://127.30.33.1:35783/ using document root <none> and password file <none>
I20260810 23:43:46.298574 2533 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:46.298694 2533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:46.299009 2533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:46.302078 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "5eac20d86cc34836862b0d86680e683a"
format_stamp: "Formatted at 2026-08-10 23:43:46 on dist-test-slave-pww0"
server_key: "bb1ed7ee2487ac547322a141fa54ec6d"
server_key_iv: "b2061361da09feeef257df869c458689"
server_key_version: "encryptionkey@0"
I20260810 23:43:46.302901 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "5eac20d86cc34836862b0d86680e683a"
format_stamp: "Formatted at 2026-08-10 23:43:46 on dist-test-slave-pww0"
server_key: "bb1ed7ee2487ac547322a141fa54ec6d"
server_key_iv: "b2061361da09feeef257df869c458689"
server_key_version: "encryptionkey@0"
I20260810 23:43:46.308459 2533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:43:46.312242 2558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:46.313694 2533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:46.313860 2533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "5eac20d86cc34836862b0d86680e683a"
format_stamp: "Formatted at 2026-08-10 23:43:46 on dist-test-slave-pww0"
server_key: "bb1ed7ee2487ac547322a141fa54ec6d"
server_key_iv: "b2061361da09feeef257df869c458689"
server_key_version: "encryptionkey@0"
I20260810 23:43:46.314070 2533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:46.342295 2533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:46.347121 2533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:46.347438 2533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:46.348565 2533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:46.350133 2533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:46.350216 2533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:46.350368 2533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:46.350422 2533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:46.382445 2533 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:34489
I20260810 23:43:46.382495 2777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:34489 every 8 connection(s)
I20260810 23:43:46.383823 2533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:43:46.389488 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 2533
I20260810 23:43:46.389662 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:43:46.390136 30852 external_mini_cluster.cc:1428] Setting key 9134fdc40ead867e59088b6bd07ec647
Aug 10 23:43:46 dist-test-slave-pww0 krb5kdc[2337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405426, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.1@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:46.417846 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:46.386625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:34343 (local address 127.30.33.62:33925)
0810 23:43:46.387112 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:46.387117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:46.388132 (+ 1015us) server_negotiation.cc:407] Connection header received
0810 23:43:46.390129 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:46.390135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:46.390235 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:46.390378 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:46.393649 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.395433 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.398170 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.399156 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.399392 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:46.405269 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:46.405277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:46.405293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:46.405393 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:46.410997 (+ 5604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:46.412756 (+ 1759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:46.412772 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:46.412777 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:46.413029 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:46.413911 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:46.413917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:46.413930 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:46.414590 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:46.415013 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:46.416817 (+ 1804us) server_negotiation.cc:299] Negotiation successful
0810 23:43:46.417278 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260810 23:43:46.418778 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:46.387154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33925 (local address 127.30.33.1:34343)
0810 23:43:46.387961 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:46.388001 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:46.389826 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:46.390666 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:46.390675 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:46.391515 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:46.393300 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:46.393330 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.395777 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.395789 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:46.397906 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:46.397931 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.399570 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.399579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:46.400022 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:46.401042 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:46.401090 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:46.404787 (+ 3697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:46.411313 (+ 6526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:46.411335 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:46.411355 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:46.412512 (+ 1157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:46.413224 (+ 712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:46.413231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:46.413234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:46.413726 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:46.414905 (+ 1179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:46.414932 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:46.415798 (+ 866us) client_negotiation.cc:769] Sending connection context
0810 23:43:46.416433 (+ 635us) client_negotiation.cc:240] Negotiation successful
0810 23:43:46.417212 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":334,"threads_started":1}
I20260810 23:43:46.420686 2778 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33925
I20260810 23:43:46.421051 2778 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:46.421866 2778 heartbeater.cc:511] Master 127.30.33.62:33925 requested a full tablet report, sending...
I20260810 23:43:46.424289 2424 ts_manager.cc:194] Registered new tserver with Master: 5eac20d86cc34836862b0d86680e683a (127.30.33.1:34489)
I20260810 23:43:46.426066 2424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.1@KRBTEST.COM'} at 127.30.33.1:34343
WARNING: no policy specified for kudu/127.30.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
Entry for principal kudu/127.30.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab.
I20260810 23:43:46.476476 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33925
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:46.835769 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:46.836082 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:46.836186 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:46.843813 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:46.843945 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:46.844070 2788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:43:46.854504 2788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.2.keytab
--principal=kudu/127.30.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:46.855911 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:46.857285 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:46.871309 2800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:46.871428 2802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:46.871397 2804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:46.872581 2788 server_base.cc:1034] running on GCE node
I20260810 23:43:46.873262 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:46.874287 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:46.875540 2788 hybrid_clock.cc:648] HybridClock initialized: now 1786405426875468 us; error 66 us; skew 500 ppm
I20260810 23:43:46.875919 2788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:46 dist-test-slave-pww0 krb5kdc[2337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405426, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:46.879148 2788 init.cc:377] Logged in from keytab as kudu/127.30.33.2@KRBTEST.COM (short username kudu)
I20260810 23:43:46.880870 2788 webserver.cc:460] Webserver started at http://127.30.33.2:36461/ using document root <none> and password file <none>
I20260810 23:43:46.881605 2788 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:46.881742 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:46.882066 2788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:46.885207 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "f6e66ce6d6ec4893b98276916ec86c02"
format_stamp: "Formatted at 2026-08-10 23:43:46 on dist-test-slave-pww0"
server_key: "744088d419c9b8c98da414d87102d85d"
server_key_iv: "175bc473d9fe53576013f5eb7e4bc70a"
server_key_version: "encryptionkey@0"
I20260810 23:43:46.886055 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "f6e66ce6d6ec4893b98276916ec86c02"
format_stamp: "Formatted at 2026-08-10 23:43:46 on dist-test-slave-pww0"
server_key: "744088d419c9b8c98da414d87102d85d"
server_key_iv: "175bc473d9fe53576013f5eb7e4bc70a"
server_key_version: "encryptionkey@0"
I20260810 23:43:46.892198 2788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:43:46.896306 2824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:46.897998 2788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:46.898211 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "f6e66ce6d6ec4893b98276916ec86c02"
format_stamp: "Formatted at 2026-08-10 23:43:46 on dist-test-slave-pww0"
server_key: "744088d419c9b8c98da414d87102d85d"
server_key_iv: "175bc473d9fe53576013f5eb7e4bc70a"
server_key_version: "encryptionkey@0"
I20260810 23:43:46.898466 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:46.919027 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:46.923936 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:46.924242 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:46.925287 2788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:46.926756 2788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:46.926838 2788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:46.926967 2788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:46.927021 2788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:46.959282 2788 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:32999
I20260810 23:43:46.959331 3029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:32999 every 8 connection(s)
I20260810 23:43:46.960714 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:43:46.962241 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 2788
I20260810 23:43:46.962445 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:43:46.963007 30852 external_mini_cluster.cc:1428] Setting key 5e6aa2fe33e392e3a78e3ef25b28f277
Aug 10 23:43:46 dist-test-slave-pww0 krb5kdc[2337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405426, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.2@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:46.984055 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:46.963807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:55335 (local address 127.30.33.62:33925)
0810 23:43:46.964273 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:46.964280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:46.967364 (+ 3084us) server_negotiation.cc:407] Connection header received
0810 23:43:46.967427 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:46.967431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:46.967545 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:46.967637 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:46.969920 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.970696 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.972988 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.973462 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.973580 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:46.977921 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:46.977927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:46.977931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:46.977973 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:46.980594 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:46.981475 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:46.981480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:46.981482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:46.981649 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:46.982181 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:46.982185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:46.982188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:46.982491 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:46.982617 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:46.983577 (+ 960us) server_negotiation.cc:299] Negotiation successful
0810 23:43:46.983786 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260810 23:43:46.985489 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:46.964275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33925 (local address 127.30.33.2:55335)
0810 23:43:46.965143 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:46.965208 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:46.967018 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:46.968001 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:46.968015 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:46.968788 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:46.969676 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:46.969706 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.970878 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.970884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:46.972775 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:46.972794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:46.973735 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:46.973741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:46.974034 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:46.974822 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:46.974860 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:46.977627 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:46.980787 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:46.980798 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:46.980817 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:46.981323 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:46.981769 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:46.981774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:46.981778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:46.982020 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:46.982628 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:46.982636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:46.983111 (+ 475us) client_negotiation.cc:769] Sending connection context
0810 23:43:46.983423 (+ 312us) client_negotiation.cc:240] Negotiation successful
0810 23:43:46.984026 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":314,"threads_started":1}
I20260810 23:43:46.987234 3030 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33925
I20260810 23:43:46.987635 3030 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:46.988557 3030 heartbeater.cc:511] Master 127.30.33.62:33925 requested a full tablet report, sending...
I20260810 23:43:46.990471 2424 ts_manager.cc:194] Registered new tserver with Master: f6e66ce6d6ec4893b98276916ec86c02 (127.30.33.2:32999)
I20260810 23:43:46.991619 2424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.2@KRBTEST.COM'} at 127.30.33.2:55335
WARNING: no policy specified for kudu/127.30.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
Entry for principal kudu/127.30.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab.
I20260810 23:43:47.044076 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33925
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:47.405671 3040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:47.405982 3040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:47.406078 3040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:47.413564 3040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:47.413705 3040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:47.413832 3040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:43:47.424217 3040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.3.keytab
--principal=kudu/127.30.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:33925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:47.425550 3040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:47.426934 3040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:43:47.429446 2778 heartbeater.cc:503] Master 127.30.33.62:33925 was elected leader, sending a full tablet report...
W20260810 23:43:47.441561 3048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:47.441650 3051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:47.441586 3049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:47.442299 3040 server_base.cc:1034] running on GCE node
I20260810 23:43:47.443054 3040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:47.444211 3040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:47.445425 3040 hybrid_clock.cc:648] HybridClock initialized: now 1786405427445368 us; error 47 us; skew 500 ppm
I20260810 23:43:47.445765 3040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:47 dist-test-slave-pww0 krb5kdc[2337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405427, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:47.449579 3040 init.cc:377] Logged in from keytab as kudu/127.30.33.3@KRBTEST.COM (short username kudu)
I20260810 23:43:47.451360 3040 webserver.cc:460] Webserver started at http://127.30.33.3:43467/ using document root <none> and password file <none>
I20260810 23:43:47.452188 3040 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:47.452312 3040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:47.452631 3040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:47.456125 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "a59efaae5956424c874f2072d2040fad"
format_stamp: "Formatted at 2026-08-10 23:43:47 on dist-test-slave-pww0"
server_key: "eb3626c463c41bb863cb6e5a490768d2"
server_key_iv: "254c549f1867c32724d2a4ad0c3547d7"
server_key_version: "encryptionkey@0"
I20260810 23:43:47.456984 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "a59efaae5956424c874f2072d2040fad"
format_stamp: "Formatted at 2026-08-10 23:43:47 on dist-test-slave-pww0"
server_key: "eb3626c463c41bb863cb6e5a490768d2"
server_key_iv: "254c549f1867c32724d2a4ad0c3547d7"
server_key_version: "encryptionkey@0"
I20260810 23:43:47.462906 3040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:43:47.467080 3062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:47.468715 3040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:47.468894 3040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "a59efaae5956424c874f2072d2040fad"
format_stamp: "Formatted at 2026-08-10 23:43:47 on dist-test-slave-pww0"
server_key: "eb3626c463c41bb863cb6e5a490768d2"
server_key_iv: "254c549f1867c32724d2a4ad0c3547d7"
server_key_version: "encryptionkey@0"
I20260810 23:43:47.469100 3040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:47.511301 3040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:47.516011 3040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:47.516350 3040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:47.517436 3040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:47.519042 3040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:47.519131 3040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:47.519268 3040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:47.519318 3040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:47.551805 3040 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:39011
I20260810 23:43:47.551849 3258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:39011 every 8 connection(s)
I20260810 23:43:47.553224 3040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:43:47.560492 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3040
I20260810 23:43:47.560704 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:43:47.561298 30852 external_mini_cluster.cc:1428] Setting key c11c0cee49ee319249e14470632d42f8
Aug 10 23:43:47 dist-test-slave-pww0 krb5kdc[2337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405427, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.3@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:43:47.580386 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.556239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:37011 (local address 127.30.33.62:33925)
0810 23:43:47.556819 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.556825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.558208 (+ 1383us) server_negotiation.cc:407] Connection header received
0810 23:43:47.561132 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.561138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.561235 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.561337 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:47.563734 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.564930 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.567327 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.567919 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.568050 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.572944 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:47.572950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:47.572953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:47.572997 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:47.576203 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:47.577392 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:47.577399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:47.577402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:47.577577 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:47.578151 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:47.578156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:47.578159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:47.578490 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:47.578632 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.579703 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.579986 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":299,"threads_started":1}
I20260810 23:43:47.581744 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.556736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:33925 (local address 127.30.33.3:37011)
0810 23:43:47.557956 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:47.558035 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:47.560633 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:47.561806 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:47.561819 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:47.562579 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:47.563496 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.563527 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.565153 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.565159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.567159 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.567176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.568186 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.568194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.568541 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.569467 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:43:47.569501 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:47.572570 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:47.576419 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:47.576431 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:47.576455 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:47.577220 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:47.577701 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:43:47.577706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:43:47.577710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:43:47.578017 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:43:47.578648 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:47.578656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:47.579175 (+ 519us) client_negotiation.cc:769] Sending connection context
0810 23:43:47.579582 (+ 407us) client_negotiation.cc:240] Negotiation successful
0810 23:43:47.580206 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":411,"threads_started":1}
I20260810 23:43:47.583213 3259 heartbeater.cc:347] Connected to a master server at 127.30.33.62:33925
I20260810 23:43:47.583586 3259 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:47.584376 3259 heartbeater.cc:511] Master 127.30.33.62:33925 requested a full tablet report, sending...
I20260810 23:43:47.585906 2424 ts_manager.cc:194] Registered new tserver with Master: a59efaae5956424c874f2072d2040fad (127.30.33.3:39011)
I20260810 23:43:47.586724 2424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.33.3@KRBTEST.COM'} at 127.30.33.3:37011
I20260810 23:43:47.594758 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:43:47.613459 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.599328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.30.33.62:33925)
0810 23:43:47.599530 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.599535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.600066 (+ 531us) server_negotiation.cc:407] Connection header received
0810 23:43:47.600232 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.600246 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.600319 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.600430 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:47.602132 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.603038 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.604576 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.604989 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.605115 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.607812 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:47.607817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:47.607821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:47.607863 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:47.610454 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:47.611254 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:47.611259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:47.611262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:47.611410 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:47.611863 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:47.611867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:47.611870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:47.612223 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:47.612485 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.612883 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.613127 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:43:47.617775 2424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:43:47.624857 2424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:43:47.657133 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.645509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.30.33.1:34489)
0810 23:43:47.646180 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.646191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.646226 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:43:47.646334 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.646342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.646639 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.648114 (+ 1475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:47.648413 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.650169 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.653196 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.654870 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.655037 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.655327 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.656571 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.656785 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":1217,"threads_started":1}
I20260810 23:43:47.658505 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.644991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34489 (local address 127.0.0.1:47808)
0810 23:43:47.645729 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:47.645776 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:47.645952 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:47.646948 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:47.646956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:47.646993 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:47.647426 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.647440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.650389 (+ 2949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.650395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.652898 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.652925 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.655533 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.655542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.655835 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.655905 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 23:43:47.656067 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:43:47.656313 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":726,"threads_started":1}
I20260810 23:43:47.663998 2649 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7350481744119a20e85cf7e752ebd (DEFAULT_TABLE table=test-table [id=d991f8e6759f42c0b434cec95d2d0ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:47.665823 2649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7350481744119a20e85cf7e752ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:47.666025 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.646575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:39011 (local address 127.0.0.1:47650)
0810 23:43:47.647125 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:47.647144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:47.647258 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:47.656315 (+ 9057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:47.656322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:47.656364 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:47.658237 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.658259 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.661038 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.661044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.662946 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.662964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.665069 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.665083 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.665320 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.665357 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:43:47.665449 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:43:47.665598 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":14,"thread_start_us":389,"threads_started":1}
I20260810 23:43:47.666167 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.647019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.30.33.3:39011)
0810 23:43:47.655423 (+ 8404us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.655430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.655498 (+ 68us) server_negotiation.cc:407] Connection header received
0810 23:43:47.655633 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.655644 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.656002 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.656555 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:47.658930 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.660609 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.663161 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.664769 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.665308 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.665576 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.665654 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.665829 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8209,"thread_start_us":8032,"threads_started":1}
I20260810 23:43:47.671810 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.647356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:32999 (local address 127.0.0.1:38648)
0810 23:43:47.658559 (+ 11203us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:47.658579 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:47.660870 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:47.661563 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:47.661568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:47.661597 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:47.662012 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.662025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.663979 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.663986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.668545 (+ 4559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.668568 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.670861 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.670869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.671046 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.671084 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:43:47.671184 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:43:47.671340 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11055,"mutex_wait_us":21,"thread_start_us":10894,"threads_started":1}
I20260810 23:43:47.670642 3148 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7350481744119a20e85cf7e752ebd (DEFAULT_TABLE table=test-table [id=d991f8e6759f42c0b434cec95d2d0ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:47.672940 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.647910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.30.33.2:32999)
0810 23:43:47.648536 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.648546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.660857 (+ 12311us) server_negotiation.cc:407] Connection header received
0810 23:43:47.661034 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.661042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.661340 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.661568 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:47.662214 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.663804 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.668797 (+ 4993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.670334 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.670471 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.670686 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.672336 (+ 1650us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.672531 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":301,"threads_started":1}
I20260810 23:43:47.673113 3148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7350481744119a20e85cf7e752ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:47.680214 2916 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7350481744119a20e85cf7e752ebd (DEFAULT_TABLE table=test-table [id=d991f8e6759f42c0b434cec95d2d0ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:47.682673 2916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7350481744119a20e85cf7e752ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:47.683665 3291 tablet_bootstrap.cc:492] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Bootstrap starting.
I20260810 23:43:47.688557 3291 tablet_bootstrap.cc:654] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:47.689639 3293 tablet_bootstrap.cc:492] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Bootstrap starting.
I20260810 23:43:47.690017 3291 log.cc:824] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:47.693678 3291 tablet_bootstrap.cc:492] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: No bootstrap required, opened a new log
I20260810 23:43:47.693661 3293 tablet_bootstrap.cc:654] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:47.694505 3291 ts_tablet_manager.cc:1397] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 23:43:47.695187 3293 log.cc:824] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:47.697763 3301 tablet_bootstrap.cc:492] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Bootstrap starting.
I20260810 23:43:47.701558 3301 tablet_bootstrap.cc:654] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:47.702908 3301 log.cc:824] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:47.705988 3301 tablet_bootstrap.cc:492] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: No bootstrap required, opened a new log
I20260810 23:43:47.706292 3301 ts_tablet_manager.cc:1397] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:43:47.716960 3301 raft_consensus.cc:348] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.717531 3301 raft_consensus.cc:374] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:47.717640 3301 raft_consensus.cc:729] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e66ce6d6ec4893b98276916ec86c02, State: Initialized, Role: FOLLOWER
I20260810 23:43:47.718605 3301 consensus_queue.cc:260] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.721237 3293 tablet_bootstrap.cc:492] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: No bootstrap required, opened a new log
I20260810 23:43:47.721611 3293 ts_tablet_manager.cc:1397] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.012s
I20260810 23:43:47.721191 3291 raft_consensus.cc:348] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.721992 3291 raft_consensus.cc:374] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:47.722136 3291 raft_consensus.cc:729] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eac20d86cc34836862b0d86680e683a, State: Initialized, Role: FOLLOWER
I20260810 23:43:47.722988 3291 consensus_queue.cc:260] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.728446 3030 heartbeater.cc:503] Master 127.30.33.62:33925 was elected leader, sending a full tablet report...
I20260810 23:43:47.729489 3301 ts_tablet_manager.cc:1428] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Time spent starting tablet: real 0.023s user 0.014s sys 0.010s
I20260810 23:43:47.730717 3293 raft_consensus.cc:348] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.731240 3293 raft_consensus.cc:374] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:47.731348 3293 raft_consensus.cc:729] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a59efaae5956424c874f2072d2040fad, State: Initialized, Role: FOLLOWER
I20260810 23:43:47.732024 3291 ts_tablet_manager.cc:1428] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Time spent starting tablet: real 0.024s user 0.016s sys 0.005s
I20260810 23:43:47.732340 3293 consensus_queue.cc:260] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.733516 3259 heartbeater.cc:503] Master 127.30.33.62:33925 was elected leader, sending a full tablet report...
I20260810 23:43:47.734231 3293 ts_tablet_manager.cc:1428] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260810 23:43:47.796917 3311 raft_consensus.cc:482] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:43:47.797333 3311 raft_consensus.cc:504] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.799366 3311 leader_election.cc:288] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eac20d86cc34836862b0d86680e683a (127.30.33.1:34489), f6e66ce6d6ec4893b98276916ec86c02 (127.30.33.2:32999)
W20260810 23:43:47.804998 3261 tablet.cc:2369] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:47.806318 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.799829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:34489 (local address 127.30.33.3:44337)
0810 23:43:47.800068 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:47.800086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:47.800239 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:47.800684 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:47.800689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:47.800720 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:47.800954 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.800966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.802190 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.802195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.804058 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.804071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.805704 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.805710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.805893 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.805921 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:43:47.806001 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:43:47.806101 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:43:47.806463 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.799858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:44337 (local address 127.30.33.1:34489)
0810 23:43:47.800101 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.800107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.800158 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:43:47.800413 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.800419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.800504 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.800613 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:47.801108 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.802013 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.804230 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.805515 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.805669 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.805786 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.806054 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.806202 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:43:47.807430 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7350481744119a20e85cf7e752ebd" candidate_uuid: "a59efaae5956424c874f2072d2040fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eac20d86cc34836862b0d86680e683a" is_pre_election: true
I20260810 23:43:47.807956 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.800590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:34481 (local address 127.30.33.2:32999)
0810 23:43:47.800816 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.800822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.801396 (+ 574us) server_negotiation.cc:407] Connection header received
0810 23:43:47.801539 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.801543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.801604 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.801696 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:47.802365 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.803572 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.805853 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.806995 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.807138 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.807239 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.807559 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.807699 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:43:47.807989 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.800304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:32999 (local address 127.30.33.3:34481)
0810 23:43:47.801283 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:47.801297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:47.801414 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:47.801763 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:47.801768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:47.801796 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:47.802182 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.802196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.803727 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.803731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.805642 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:47.805658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.807217 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.807223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:47.807373 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.807398 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:43:47.807526 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:43:47.807663 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":662,"threads_started":1}
I20260810 23:43:47.808009 2688 raft_consensus.cc:2454] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a59efaae5956424c874f2072d2040fad in term 0.
I20260810 23:43:47.808768 3065 leader_election.cc:302] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eac20d86cc34836862b0d86680e683a, a59efaae5956424c874f2072d2040fad; no voters:
I20260810 23:43:47.809186 3311 raft_consensus.cc:2781] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:43:47.809104 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7350481744119a20e85cf7e752ebd" candidate_uuid: "a59efaae5956424c874f2072d2040fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e66ce6d6ec4893b98276916ec86c02" is_pre_election: true
I20260810 23:43:47.809351 3311 raft_consensus.cc:482] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:43:47.809458 3311 raft_consensus.cc:3037] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:47.809713 2949 raft_consensus.cc:2454] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a59efaae5956424c874f2072d2040fad in term 0.
I20260810 23:43:47.811969 3311 raft_consensus.cc:504] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.812601 3311 leader_election.cc:288] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [CANDIDATE]: Term 1 election: Requested vote from peers 5eac20d86cc34836862b0d86680e683a (127.30.33.1:34489), f6e66ce6d6ec4893b98276916ec86c02 (127.30.33.2:32999)
I20260810 23:43:47.813100 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7350481744119a20e85cf7e752ebd" candidate_uuid: "a59efaae5956424c874f2072d2040fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eac20d86cc34836862b0d86680e683a"
I20260810 23:43:47.813145 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7350481744119a20e85cf7e752ebd" candidate_uuid: "a59efaae5956424c874f2072d2040fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e66ce6d6ec4893b98276916ec86c02"
I20260810 23:43:47.813347 2688 raft_consensus.cc:3037] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:47.813347 2949 raft_consensus.cc:3037] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:47.816715 2688 raft_consensus.cc:2454] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a59efaae5956424c874f2072d2040fad in term 1.
I20260810 23:43:47.816718 2949 raft_consensus.cc:2454] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a59efaae5956424c874f2072d2040fad in term 1.
I20260810 23:43:47.817250 3065 leader_election.cc:302] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a59efaae5956424c874f2072d2040fad, f6e66ce6d6ec4893b98276916ec86c02; no voters:
I20260810 23:43:47.817561 3311 raft_consensus.cc:2781] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:47.818055 3311 raft_consensus.cc:686] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 1 LEADER]: Becoming Leader. State: Replica: a59efaae5956424c874f2072d2040fad, State: Running, Role: LEADER
I20260810 23:43:47.818642 3311 consensus_queue.cc:237] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } }
I20260810 23:43:47.823562 2420 catalog_manager.cc:5495] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad reported cstate change: term changed from 0 to 1, leader changed from <none> to a59efaae5956424c874f2072d2040fad (127.30.33.3). New cstate: current_term: 1 leader_uuid: "a59efaae5956424c874f2072d2040fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a59efaae5956424c874f2072d2040fad" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 39011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 } health_report { overall_health: UNKNOWN } } }
I20260810 23:43:47.839996 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.834413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.30.33.3:39011)
0810 23:43:47.834588 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.834593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.834702 (+ 109us) server_negotiation.cc:407] Connection header received
0810 23:43:47.834901 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.834905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.835001 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.835142 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:43:47.835508 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.836410 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.837873 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.838312 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.838417 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.838942 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:43:47.839525 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:43:47.839636 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.839715 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.839790 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:43:47.848784 2949 raft_consensus.cc:1261] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 1 FOLLOWER]: Refusing update from remote peer a59efaae5956424c874f2072d2040fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:47.848805 2688 raft_consensus.cc:1261] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 1 FOLLOWER]: Refusing update from remote peer a59efaae5956424c874f2072d2040fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:47.849932 3314 consensus_queue.cc:1035] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eac20d86cc34836862b0d86680e683a" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:47.850354 3311 consensus_queue.cc:1035] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e66ce6d6ec4893b98276916ec86c02" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 32999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:47.865463 3318 mvcc.cc:205] Tried to move back new op lower bound from 7317116632457162752 to 7317116632349556736. Current Snapshot: MvccSnapshot[applied={T|T < 7317116632457162752}]
I20260810 23:43:47.883776 2424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:47.884172 2424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:47.888717 2424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6ea16aa8c684dde8a854fbd06d729e9: Sending DeleteTablet for 3 replicas of tablet 1fc7350481744119a20e85cf7e752ebd
I20260810 23:43:47.890043 2649 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc7350481744119a20e85cf7e752ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:47 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:47808
I20260810 23:43:47.890187 3148 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc7350481744119a20e85cf7e752ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:47 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:47650
I20260810 23:43:47.890960 2916 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc7350481744119a20e85cf7e752ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:47 UTC) from {username='kudu', principal='kudu/127.30.33.62@KRBTEST.COM'} at 127.0.0.1:38648
I20260810 23:43:47.891860 3331 tablet_replica.cc:330] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: stopping tablet replica
I20260810 23:43:47.892470 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 2533
I20260810 23:43:47.892436 3331 raft_consensus.cc:2229] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:43:47.893373 3331 raft_consensus.cc:2258] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:47.891287 3330 tablet_replica.cc:330] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: stopping tablet replica
I20260810 23:43:47.895884 3330 raft_consensus.cc:2229] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:47.896517 3330 raft_consensus.cc:2258] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:47.897481 3332 tablet_replica.cc:330] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: stopping tablet replica
I20260810 23:43:47.897953 3332 raft_consensus.cc:2229] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:47.898521 3332 raft_consensus.cc:2258] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:47.898854 3331 ts_tablet_manager.cc:1905] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:47.899312 3330 ts_tablet_manager.cc:1905] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:47.904553 3332 ts_tablet_manager.cc:1905] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:43:47 dist-test-slave-pww0 krb5kdc[2337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.1@KRBTEST.COM
I20260810 23:43:47.906715 3330 ts_tablet_manager.cc:1918] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:47.907012 3330 log.cc:1196] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-0/wal/wals/1fc7350481744119a20e85cf7e752ebd
I20260810 23:43:47.907701 3330 ts_tablet_manager.cc:1939] T 1fc7350481744119a20e85cf7e752ebd P 5eac20d86cc34836862b0d86680e683a: Deleting consensus metadata
I20260810 23:43:47.909905 2392 catalog_manager.cc:4841] TS 5eac20d86cc34836862b0d86680e683a (127.30.33.1:34489): tablet 1fc7350481744119a20e85cf7e752ebd (table test-table [id=d991f8e6759f42c0b434cec95d2d0ab7]) successfully deleted
I20260810 23:43:47.913136 3331 ts_tablet_manager.cc:1918] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:47.913393 3331 log.cc:1196] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/1fc7350481744119a20e85cf7e752ebd
I20260810 23:43:47.914068 3331 ts_tablet_manager.cc:1939] T 1fc7350481744119a20e85cf7e752ebd P a59efaae5956424c874f2072d2040fad: Deleting consensus metadata
I20260810 23:43:47.914108 3332 ts_tablet_manager.cc:1918] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:47.914340 3332 log.cc:1196] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/1fc7350481744119a20e85cf7e752ebd
I20260810 23:43:47.914932 3332 ts_tablet_manager.cc:1939] T 1fc7350481744119a20e85cf7e752ebd P f6e66ce6d6ec4893b98276916ec86c02: Deleting consensus metadata
I20260810 23:43:47.916162 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:47.894756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.30.33.1:34489)
0810 23:43:47.894972 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:47.894979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:47.895540 (+ 561us) server_negotiation.cc:407] Connection header received
0810 23:43:47.895869 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:47.895876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:47.895989 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:47.896090 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:47.898293 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.899389 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.901306 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:47.901965 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:47.902079 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:47.908083 (+ 6004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:47.908100 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:47.908119 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:47.908161 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:47.912062 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:47.912991 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:47.913001 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:47.913004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:47.913198 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:47.913876 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:47.913882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:47.913885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:47.914508 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:47.914793 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:47.915511 (+ 718us) server_negotiation.cc:299] Negotiation successful
0810 23:43:47.915773 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:43:47.916158 2392 catalog_manager.cc:4841] TS a59efaae5956424c874f2072d2040fad (127.30.33.3:39011): tablet 1fc7350481744119a20e85cf7e752ebd (table test-table [id=d991f8e6759f42c0b434cec95d2d0ab7]) successfully deleted
I20260810 23:43:47.916818 2392 catalog_manager.cc:4841] TS f6e66ce6d6ec4893b98276916ec86c02 (127.30.33.2:32999): tablet 1fc7350481744119a20e85cf7e752ebd (table test-table [id=d991f8e6759f42c0b434cec95d2d0ab7]) successfully deleted
I20260810 23:43:47.918219 2766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:48.052737 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 2533
I20260810 23:43:48.106351 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 2788
Aug 10 23:43:48 dist-test-slave-pww0 krb5kdc[2337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.2@KRBTEST.COM
I20260810 23:43:48.122996 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:48.107326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.30.33.2:32999)
0810 23:43:48.107562 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:48.107568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:48.107589 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:43:48.107737 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:48.107741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:48.107813 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:48.107898 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:48.109258 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:48.110033 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:48.111405 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:48.111919 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:48.112040 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:48.116590 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:48.116610 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:48.116635 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:48.116689 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:48.119857 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:48.120706 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:48.120715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:48.120718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:48.120859 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:48.121273 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:48.121277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:48.121279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:48.121755 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:48.122055 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:48.122400 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 23:43:48.122638 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:43:48.123697 3021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:48.215286 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 2788
I20260810 23:43:48.240068 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3040
Aug 10 23:43:48 dist-test-slave-pww0 krb5kdc[2337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.3@KRBTEST.COM
I20260810 23:43:48.263259 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:48.240977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.30.33.3:39011)
0810 23:43:48.241165 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:48.241170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:48.241194 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:43:48.241413 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:48.241417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:48.241495 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:48.241580 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:48.243568 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:48.244431 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:48.246520 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:48.247089 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:48.247209 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:48.256529 (+ 9320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:48.256551 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:48.256569 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:48.256621 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:48.260011 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:48.260881 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:48.260890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:48.260893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:48.261068 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:48.261519 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:48.261524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:48.261526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:48.261936 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:48.262307 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:48.262649 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:43:48.262885 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:43:48.264117 3249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:48.391090 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3040
I20260810 23:43:48.418162 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 2356
I20260810 23:43:48.419560 2484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:48.531587 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 2356
2026-08-10T23:43:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4969 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T23:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:48Z Disabled control of system clock
I20260810 23:43:48.633957 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36383
--webserver_interface=127.30.33.62
--webserver_port=0
--builtin_ntp_servers=127.30.33.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:36383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:43:48.992029 3351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:48.992352 3351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:48.992436 3351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:48.999939 3351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:49.000069 3351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:49.000108 3351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:49.000166 3351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:49.000211 3351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:49.009824 3351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:36383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:36383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:49.011193 3351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:49.013065 3351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:49.025439 3361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:49.025372 3365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:49.025398 3367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:49.026733 3351 server_base.cc:1034] running on GCE node
I20260810 23:43:49.027623 3351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:49.029151 3351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:49.030356 3351 hybrid_clock.cc:648] HybridClock initialized: now 1786405429030301 us; error 57 us; skew 500 ppm
I20260810 23:43:49.030728 3351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:49.033227 3351 webserver.cc:460] Webserver started at http://127.30.33.62:43063/ using document root <none> and password file <none>
I20260810 23:43:49.033962 3351 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:49.034061 3351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:49.034397 3351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:49.037657 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "b17ff1e88e58431c98e47f4640e04e29"
format_stamp: "Formatted at 2026-08-10 23:43:49 on dist-test-slave-pww0"
server_key: "88c86148fc71e63934a0a4503fc658a7"
server_key_iv: "d2191d6a15fa6888e4189cb19c471eaf"
server_key_version: "encryptionkey@0"
I20260810 23:43:49.038550 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "b17ff1e88e58431c98e47f4640e04e29"
format_stamp: "Formatted at 2026-08-10 23:43:49 on dist-test-slave-pww0"
server_key: "88c86148fc71e63934a0a4503fc658a7"
server_key_iv: "d2191d6a15fa6888e4189cb19c471eaf"
server_key_version: "encryptionkey@0"
I20260810 23:43:49.044435 3351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:43:49.048264 3383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:49.049753 3351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:49.049947 3351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "b17ff1e88e58431c98e47f4640e04e29"
format_stamp: "Formatted at 2026-08-10 23:43:49 on dist-test-slave-pww0"
server_key: "88c86148fc71e63934a0a4503fc658a7"
server_key_iv: "d2191d6a15fa6888e4189cb19c471eaf"
server_key_version: "encryptionkey@0"
I20260810 23:43:49.050168 3351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:49.066394 3351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:49.067420 3351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:49.067786 3351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:49.088193 3351 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:36383
I20260810 23:43:49.088222 3469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:36383 every 8 connection(s)
I20260810 23:43:49.089666 3351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:49.093803 3470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:49.098289 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3351
I20260810 23:43:49.098491 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:49.099032 30852 external_mini_cluster.cc:1428] Setting key a2e24b62d65bcc131e8a8e7a15ec728d
I20260810 23:43:49.106208 3470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29: Bootstrap starting.
I20260810 23:43:49.109547 3470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:49.110662 3470 log.cc:824] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:49.114246 3470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29: No bootstrap required, opened a new log
I20260810 23:43:49.114420 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:49.101194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.30.33.62:36383)
0810 23:43:49.102317 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:49.102329 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:49.102376 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:43:49.104853 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:49.104882 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:49.105431 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:49.105842 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:49.106474 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:49.107905 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:49.109310 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:49.110113 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:49.110322 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:49.111548 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:49.111557 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:49.111575 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:49.112234 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:49.112344 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:49.112379 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:49.112507 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:49.112775 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:43:49.113120 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":400,"threads_started":1}
I20260810 23:43:49.121891 3470 raft_consensus.cc:348] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17ff1e88e58431c98e47f4640e04e29" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36383 } }
I20260810 23:43:49.122315 3470 raft_consensus.cc:374] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:49.122402 3470 raft_consensus.cc:729] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17ff1e88e58431c98e47f4640e04e29, State: Initialized, Role: FOLLOWER
I20260810 23:43:49.122944 3470 consensus_queue.cc:260] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17ff1e88e58431c98e47f4640e04e29" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36383 } }
I20260810 23:43:49.123189 3470 raft_consensus.cc:388] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:49.123330 3470 raft_consensus.cc:482] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:49.123553 3470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:49.126928 3470 raft_consensus.cc:504] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17ff1e88e58431c98e47f4640e04e29" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36383 } }
I20260810 23:43:49.127651 3470 leader_election.cc:302] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b17ff1e88e58431c98e47f4640e04e29; no voters:
I20260810 23:43:49.128386 3470 leader_election.cc:288] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:49.128531 3480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:49.130903 3480 raft_consensus.cc:686] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [term 1 LEADER]: Becoming Leader. State: Replica: b17ff1e88e58431c98e47f4640e04e29, State: Running, Role: LEADER
I20260810 23:43:49.131131 3470 sys_catalog.cc:564] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:49.131592 3480 consensus_queue.cc:237] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17ff1e88e58431c98e47f4640e04e29" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36383 } }
I20260810 23:43:49.134864 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b17ff1e88e58431c98e47f4640e04e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17ff1e88e58431c98e47f4640e04e29" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36383 } } }
I20260810 23:43:49.134836 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b17ff1e88e58431c98e47f4640e04e29. Latest consensus state: current_term: 1 leader_uuid: "b17ff1e88e58431c98e47f4640e04e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17ff1e88e58431c98e47f4640e04e29" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 36383 } } }
I20260810 23:43:49.135432 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:49.135435 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:49.140897 3494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:49.146714 3494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:49.158130 3494 catalog_manager.cc:1347] Generated new cluster ID: 4004adf9d60044589f3bbf7d14bc025f
I20260810 23:43:49.158285 3494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:49.195155 3494 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:49.197149 3494 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:49.214495 3494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29: Generated new TSK 0
I20260810 23:43:49.215732 3494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:43:49.236508 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.1:0
--local_ip_for_outbound_sockets=127.30.33.1
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36383
--builtin_ntp_servers=127.30.33.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:43:49.595377 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:49.595779 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:49.595899 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:49.603852 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:49.603973 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:49.604111 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.1
I20260810 23:43:49.613818 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.33.1
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.1
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:49.615273 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:49.617165 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:49.630788 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:49.630740 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:49.630761 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:49.631529 3511 server_base.cc:1034] running on GCE node
I20260810 23:43:49.632284 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:49.633351 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:49.634594 3511 hybrid_clock.cc:648] HybridClock initialized: now 1786405429634539 us; error 50 us; skew 500 ppm
I20260810 23:43:49.634953 3511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:49.637600 3511 webserver.cc:460] Webserver started at http://127.30.33.1:42409/ using document root <none> and password file <none>
I20260810 23:43:49.638360 3511 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:49.638471 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:49.638793 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:49.642074 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data/instance:
uuid: "3724716d66b6466ab9ccadb198e9dd38"
format_stamp: "Formatted at 2026-08-10 23:43:49 on dist-test-slave-pww0"
server_key: "373862298f4719e4f7e6c81274c5bb02"
server_key_iv: "34cec53e68c0e790b2eff99f05949718"
server_key_version: "encryptionkey@0"
I20260810 23:43:49.642917 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance:
uuid: "3724716d66b6466ab9ccadb198e9dd38"
format_stamp: "Formatted at 2026-08-10 23:43:49 on dist-test-slave-pww0"
server_key: "373862298f4719e4f7e6c81274c5bb02"
server_key_iv: "34cec53e68c0e790b2eff99f05949718"
server_key_version: "encryptionkey@0"
I20260810 23:43:49.648666 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:43:49.652738 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:49.654366 3511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:43:49.654542 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/wal
uuid: "3724716d66b6466ab9ccadb198e9dd38"
format_stamp: "Formatted at 2026-08-10 23:43:49 on dist-test-slave-pww0"
server_key: "373862298f4719e4f7e6c81274c5bb02"
server_key_iv: "34cec53e68c0e790b2eff99f05949718"
server_key_version: "encryptionkey@0"
I20260810 23:43:49.654816 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:49.668457 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:49.669346 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:49.669618 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:49.670730 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:49.672362 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:49.672454 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:49.672561 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:49.672632 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:49.702854 3511 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.1:43817
I20260810 23:43:49.702893 3736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.1:43817 every 8 connection(s)
I20260810 23:43:49.704219 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/data/info.pb
I20260810 23:43:49.712908 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3511
I20260810 23:43:49.713073 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-0/wal/instance
I20260810 23:43:49.713516 30852 external_mini_cluster.cc:1428] Setting key 1d124803a56d33ceddcce2385eef9128
I20260810 23:43:49.716804 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.2:0
--local_ip_for_outbound_sockets=127.30.33.2
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36383
--builtin_ntp_servers=127.30.33.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:43:49.717630 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:49.706747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:36489 (local address 127.30.33.62:36383)
0810 23:43:49.707250 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:49.707257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:49.708559 (+ 1302us) server_negotiation.cc:407] Connection header received
0810 23:43:49.710405 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:49.710409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:49.710492 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:49.710580 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:49.711776 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:49.712535 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:49.714435 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:49.714996 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:49.715104 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:49.716689 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:49.716696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:49.716700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:49.716724 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:49.716787 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:49.716806 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:49.716907 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:49.717148 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:43:49.717250 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":231,"threads_started":1}
I20260810 23:43:49.718588 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:49.707379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36383 (local address 127.30.33.1:36489)
0810 23:43:49.708390 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:49.708428 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:49.710108 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:49.710791 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:49.710800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:49.710894 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:49.711600 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:49.711624 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:49.712705 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:49.712710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:49.714286 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:49.714302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:49.715233 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:49.715242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:49.715572 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:49.716338 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:49.716341 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:49.716389 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:49.716475 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:49.716901 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:49.716911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:49.716919 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:43:49.717092 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 23:43:49.717484 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":287,"threads_started":1}
I20260810 23:43:49.720208 3739 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36383
I20260810 23:43:49.720510 3739 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:49.721272 3739 heartbeater.cc:511] Master 127.30.33.62:36383 requested a full tablet report, sending...
I20260810 23:43:49.723290 3413 ts_manager.cc:194] Registered new tserver with Master: 3724716d66b6466ab9ccadb198e9dd38 (127.30.33.1:43817)
I20260810 23:43:49.724797 3413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.1:36489
W20260810 23:43:50.092892 3746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:50.093276 3746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:50.093447 3746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:50.104781 3746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:50.104933 3746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:50.105132 3746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.2
I20260810 23:43:50.115252 3746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.33.2
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.2
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:50.116649 3746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:50.118469 3746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:50.132063 3756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:50.132133 3758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:50.132158 3763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:50.133252 3746 server_base.cc:1034] running on GCE node
I20260810 23:43:50.133967 3746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:50.135003 3746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:50.136240 3746 hybrid_clock.cc:648] HybridClock initialized: now 1786405430136175 us; error 55 us; skew 500 ppm
I20260810 23:43:50.136619 3746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:50.139390 3746 webserver.cc:460] Webserver started at http://127.30.33.2:39249/ using document root <none> and password file <none>
I20260810 23:43:50.140255 3746 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:50.140362 3746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:50.140733 3746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:50.144133 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data/instance:
uuid: "d1dbde6deec74b4ebc3060bc1093631b"
format_stamp: "Formatted at 2026-08-10 23:43:50 on dist-test-slave-pww0"
server_key: "c1d8b08af09b06234f1c9471d8e05697"
server_key_iv: "0ab8261a979ac9925a52933b716c96ae"
server_key_version: "encryptionkey@0"
I20260810 23:43:50.145020 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance:
uuid: "d1dbde6deec74b4ebc3060bc1093631b"
format_stamp: "Formatted at 2026-08-10 23:43:50 on dist-test-slave-pww0"
server_key: "c1d8b08af09b06234f1c9471d8e05697"
server_key_iv: "0ab8261a979ac9925a52933b716c96ae"
server_key_version: "encryptionkey@0"
I20260810 23:43:50.151098 3746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 23:43:50.155309 3773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:50.156922 3746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:50.157110 3746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal
uuid: "d1dbde6deec74b4ebc3060bc1093631b"
format_stamp: "Formatted at 2026-08-10 23:43:50 on dist-test-slave-pww0"
server_key: "c1d8b08af09b06234f1c9471d8e05697"
server_key_iv: "0ab8261a979ac9925a52933b716c96ae"
server_key_version: "encryptionkey@0"
I20260810 23:43:50.157366 3746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:50.182144 3746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:50.183076 3746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:50.183372 3746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:50.184531 3746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:50.186225 3746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:50.186334 3746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:50.186451 3746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:50.186525 3746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:50.217985 3746 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.2:40669
I20260810 23:43:50.218016 3971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.2:40669 every 8 connection(s)
I20260810 23:43:50.219367 3746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/data/info.pb
I20260810 23:43:50.224642 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3746
I20260810 23:43:50.224825 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal/instance
I20260810 23:43:50.225266 30852 external_mini_cluster.cc:1428] Setting key ebf29aa0dab12c096536be5bf2ca7cbd
I20260810 23:43:50.228407 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.33.3:0
--local_ip_for_outbound_sockets=127.30.33.3
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36383
--builtin_ntp_servers=127.30.33.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:43:50.233807 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.222130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.2:42957 (local address 127.30.33.62:36383)
0810 23:43:50.222566 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:50.222571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:50.223959 (+ 1388us) server_negotiation.cc:407] Connection header received
0810 23:43:50.226032 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:50.226038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:50.226151 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:50.226288 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:50.227546 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.228274 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.230667 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.231131 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.231227 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.233005 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:50.233012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:50.233015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:50.233039 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:50.233082 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:50.233094 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:50.233167 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:50.233413 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 23:43:50.233510 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":218,"threads_started":1}
I20260810 23:43:50.235289 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.222775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36383 (local address 127.30.33.2:42957)
0810 23:43:50.223797 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:50.223838 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:50.225713 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:50.226557 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:50.226567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:50.226710 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:50.227346 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.227365 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.228461 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.228468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.230471 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.230491 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.231388 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.231395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.231788 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.232596 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:50.232600 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:50.232661 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:50.232795 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:50.233184 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:50.233195 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:50.233203 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:43:50.233395 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 23:43:50.233860 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":263,"threads_started":1}
I20260810 23:43:50.236928 3972 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36383
I20260810 23:43:50.237308 3972 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:50.238288 3972 heartbeater.cc:511] Master 127.30.33.62:36383 requested a full tablet report, sending...
I20260810 23:43:50.240085 3413 ts_manager.cc:194] Registered new tserver with Master: d1dbde6deec74b4ebc3060bc1093631b (127.30.33.2:40669)
I20260810 23:43:50.241077 3413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.2:42957
W20260810 23:43:50.587157 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:50.587481 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:50.587618 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:50.595036 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:50.595158 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:50.595302 3979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.33.3
I20260810 23:43:50.604863 3979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.33.3
--webserver_port=0
--tserver_master_addrs=127.30.33.62:36383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.33.3
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:50.606254 3979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:50.608161 3979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:50.621488 3986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:50.621438 3989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:50.621984 3985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:50.622462 3979 server_base.cc:1034] running on GCE node
I20260810 23:43:50.623095 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:50.624183 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:50.625396 3979 hybrid_clock.cc:648] HybridClock initialized: now 1786405430625338 us; error 52 us; skew 500 ppm
I20260810 23:43:50.625716 3979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:43:50.628237 3979 webserver.cc:460] Webserver started at http://127.30.33.3:34747/ using document root <none> and password file <none>
I20260810 23:43:50.628988 3979 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:50.629101 3979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:50.629420 3979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:50.632632 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data/instance:
uuid: "e3b30e06dba14718b0fb5d4aacf6d81a"
format_stamp: "Formatted at 2026-08-10 23:43:50 on dist-test-slave-pww0"
server_key: "165ba218502e97df4c735adf6b5bc9e6"
server_key_iv: "0c2877b50dbab1331530f07c68b2d14e"
server_key_version: "encryptionkey@0"
I20260810 23:43:50.633499 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance:
uuid: "e3b30e06dba14718b0fb5d4aacf6d81a"
format_stamp: "Formatted at 2026-08-10 23:43:50 on dist-test-slave-pww0"
server_key: "165ba218502e97df4c735adf6b5bc9e6"
server_key_iv: "0c2877b50dbab1331530f07c68b2d14e"
server_key_version: "encryptionkey@0"
I20260810 23:43:50.639155 3979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:43:50.643083 3999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:50.644690 3979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:43:50.644897 3979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal
uuid: "e3b30e06dba14718b0fb5d4aacf6d81a"
format_stamp: "Formatted at 2026-08-10 23:43:50 on dist-test-slave-pww0"
server_key: "165ba218502e97df4c735adf6b5bc9e6"
server_key_iv: "0c2877b50dbab1331530f07c68b2d14e"
server_key_version: "encryptionkey@0"
I20260810 23:43:50.645138 3979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:50.682668 3979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:50.683678 3979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:50.683982 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:50.685050 3979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:43:50.686666 3979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:43:50.686775 3979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:50.686883 3979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:43:50.686954 3979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:50.721657 3979 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.3:43697
I20260810 23:43:50.721745 4210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.3:43697 every 8 connection(s)
I20260810 23:43:50.723032 3979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/data/info.pb
I20260810 23:43:50.726279 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 3979
I20260810 23:43:50.726435 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal/instance
I20260810 23:43:50.727046 30852 external_mini_cluster.cc:1428] Setting key 3c7188327a04bdf5665970f54171e3cc
I20260810 23:43:50.727912 3739 heartbeater.cc:503] Master 127.30.33.62:36383 was elected leader, sending a full tablet report...
I20260810 23:43:50.736407 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.725792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.3:60613 (local address 127.30.33.62:36383)
0810 23:43:50.726016 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:50.726023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:50.727765 (+ 1742us) server_negotiation.cc:407] Connection header received
0810 23:43:50.729963 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:50.729968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:50.730053 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:50.730143 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:50.731483 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.732317 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.733878 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.734271 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.734370 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.735702 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:50.735707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:50.735710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:50.735753 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:50.735783 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:50.735795 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:50.735861 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:50.736026 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 23:43:50.736108 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:43:50.737526 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.726361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:36383 (local address 127.30.33.3:60613)
0810 23:43:50.727546 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:50.727605 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:50.729605 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:50.730380 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:50.730388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:50.730485 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:43:50.731310 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.731334 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.732452 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.732456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.733733 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.733747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.734476 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.734481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.734691 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.735374 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:43:50.735376 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:43:50.735418 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:43:50.735549 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:43:50.735865 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:43:50.735870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:43:50.735876 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:43:50.736015 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:43:50.736348 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260810 23:43:50.739020 4211 heartbeater.cc:347] Connected to a master server at 127.30.33.62:36383
I20260810 23:43:50.739346 4211 heartbeater.cc:464] Registering TS with master...
I20260810 23:43:50.740206 4211 heartbeater.cc:511] Master 127.30.33.62:36383 requested a full tablet report, sending...
I20260810 23:43:50.741743 3412 ts_manager.cc:194] Registered new tserver with Master: e3b30e06dba14718b0fb5d4aacf6d81a (127.30.33.3:43697)
I20260810 23:43:50.742833 3412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.33.3:60613
I20260810 23:43:50.743564 30852 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:43:50.753697 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.747742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.30.33.62:36383)
0810 23:43:50.747960 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:50.747968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:50.748348 (+ 380us) server_negotiation.cc:407] Connection header received
0810 23:43:50.748489 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:50.748492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:50.748560 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:50.748644 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:50.749145 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.749940 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.751300 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.751781 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.751901 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.752980 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:50.752986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:50.752989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:50.753009 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:50.753042 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:50.753054 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:50.753131 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:50.753277 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 23:43:50.753375 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:43:50.757743 3412 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:43:50.765120 3412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:43:50.791821 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.778315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.30.33.1:43817)
0810 23:43:50.784029 (+ 5714us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:50.784036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:50.784061 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:43:50.784168 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:50.784177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:50.784442 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:50.784645 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:50.785246 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.786672 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.788994 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.790539 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.790700 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.791018 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:50.791218 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 23:43:50.791420 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5532,"thread_start_us":5390,"threads_started":1}
I20260810 23:43:50.791867 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.778100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34400)
0810 23:43:50.779723 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:50.779785 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:50.779984 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:50.784662 (+ 4678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:50.784667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:50.784694 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:50.785045 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.785059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.786817 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.786826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.788780 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.788794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.790743 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.790749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.790949 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.791006 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 23:43:50.791176 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 23:43:50.791397 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":70,"thread_start_us":2144,"threads_started":1}
I20260810 23:43:50.791952 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.778212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:40669 (local address 127.0.0.1:57970)
0810 23:43:50.780626 (+ 2414us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:50.780645 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:50.780899 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:50.783628 (+ 2729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:50.783637 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:50.783674 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:50.784104 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.784114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.786458 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.786463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.788986 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.789005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.791118 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.791124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.791308 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.791343 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:43:50.791432 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:43:50.791589 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"mutex_wait_us":75,"thread_start_us":980,"threads_started":1}
I20260810 23:43:50.792608 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.781430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43697 (local address 127.0.0.1:35356)
0810 23:43:50.783759 (+ 2329us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:50.783786 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:50.783929 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:50.785243 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:50.785248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:50.785272 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:50.785645 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.785655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.787378 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.787383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.789599 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:50.789614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.791850 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.791857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:50.792038 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.792070 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:43:50.792153 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:43:50.792281 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2182,"mutex_wait_us":6,"thread_start_us":1988,"threads_started":1}
I20260810 23:43:50.793073 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.778795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.30.33.2:40669)
0810 23:43:50.782792 (+ 3997us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:50.782802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:50.782839 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:43:50.782958 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:50.782967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:50.783277 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:50.784295 (+ 1018us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:50.784505 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.786309 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.789204 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.790614 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.790744 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.790983 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:50.792423 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0810 23:43:50.792623 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3781,"thread_start_us":3631,"threads_started":1}
I20260810 23:43:50.793249 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:50.782074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.30.33.3:43697)
0810 23:43:50.783254 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:50.783263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:50.784066 (+ 803us) server_negotiation.cc:407] Connection header received
0810 23:43:50.784168 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:50.784176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:50.784442 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:50.784650 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:50.785876 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.787211 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.789848 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:50.791275 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:50.791396 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:50.791701 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:50.792603 (+ 902us) server_negotiation.cc:299] Negotiation successful
0810 23:43:50.792907 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":856,"threads_started":1}
I20260810 23:43:50.796480 3626 tablet_service.cc:1467] Processing CreateTablet for tablet ec3a5e2342214402a455bb95c6dc3c87 (DEFAULT_TABLE table=test-table [id=4711887444e44ea694f1df7c317b2f7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:50.796885 3852 tablet_service.cc:1467] Processing CreateTablet for tablet ec3a5e2342214402a455bb95c6dc3c87 (DEFAULT_TABLE table=test-table [id=4711887444e44ea694f1df7c317b2f7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:50.796883 4100 tablet_service.cc:1467] Processing CreateTablet for tablet ec3a5e2342214402a455bb95c6dc3c87 (DEFAULT_TABLE table=test-table [id=4711887444e44ea694f1df7c317b2f7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:43:50.797935 3626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3a5e2342214402a455bb95c6dc3c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:50.798341 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3a5e2342214402a455bb95c6dc3c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:50.798341 4100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3a5e2342214402a455bb95c6dc3c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:50.810292 4249 tablet_bootstrap.cc:492] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: Bootstrap starting.
I20260810 23:43:50.813441 4254 tablet_bootstrap.cc:492] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Bootstrap starting.
I20260810 23:43:50.814303 4250 tablet_bootstrap.cc:492] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Bootstrap starting.
I20260810 23:43:50.815595 4249 tablet_bootstrap.cc:654] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:50.817003 4249 log.cc:824] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:50.817967 4254 tablet_bootstrap.cc:654] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:50.818806 4250 tablet_bootstrap.cc:654] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:50.819139 4254 log.cc:824] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:50.820053 4250 log.cc:824] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:50.820581 4249 tablet_bootstrap.cc:492] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: No bootstrap required, opened a new log
I20260810 23:43:50.820930 4249 ts_tablet_manager.cc:1397] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260810 23:43:50.822033 4254 tablet_bootstrap.cc:492] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: No bootstrap required, opened a new log
I20260810 23:43:50.822310 4254 ts_tablet_manager.cc:1397] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 23:43:50.823155 4250 tablet_bootstrap.cc:492] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: No bootstrap required, opened a new log
I20260810 23:43:50.823516 4250 ts_tablet_manager.cc:1397] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:43:50.829959 4254 raft_consensus.cc:348] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:50.830390 4254 raft_consensus.cc:374] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:50.830510 4254 raft_consensus.cc:729] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b30e06dba14718b0fb5d4aacf6d81a, State: Initialized, Role: FOLLOWER
I20260810 23:43:50.831213 4254 consensus_queue.cc:260] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:50.832473 4211 heartbeater.cc:503] Master 127.30.33.62:36383 was elected leader, sending a full tablet report...
I20260810 23:43:50.833034 4254 ts_tablet_manager.cc:1428] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260810 23:43:50.833881 4250 raft_consensus.cc:348] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:50.833881 4249 raft_consensus.cc:348] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:50.834357 4249 raft_consensus.cc:374] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:50.834362 4250 raft_consensus.cc:374] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:50.834461 4250 raft_consensus.cc:729] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1dbde6deec74b4ebc3060bc1093631b, State: Initialized, Role: FOLLOWER
I20260810 23:43:50.834461 4249 raft_consensus.cc:729] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3724716d66b6466ab9ccadb198e9dd38, State: Initialized, Role: FOLLOWER
I20260810 23:43:50.835234 4249 consensus_queue.cc:260] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:50.835222 4250 consensus_queue.cc:260] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:50.836938 3972 heartbeater.cc:503] Master 127.30.33.62:36383 was elected leader, sending a full tablet report...
I20260810 23:43:50.837531 4250 ts_tablet_manager.cc:1428] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260810 23:43:50.838824 4249 ts_tablet_manager.cc:1428] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: Time spent starting tablet: real 0.018s user 0.010s sys 0.003s
W20260810 23:43:50.956712 3741 tablet.cc:2369] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:43:50.971572 3974 tablet.cc:2369] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:43:50.974771 4213 tablet.cc:2369] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:43:51.047991 4268 raft_consensus.cc:482] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:43:51.048379 4268 raft_consensus.cc:504] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:51.050196 4268 leader_election.cc:288] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1dbde6deec74b4ebc3060bc1093631b (127.30.33.2:40669), e3b30e06dba14718b0fb5d4aacf6d81a (127.30.33.3:43697)
I20260810 23:43:51.058465 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.051296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.3:43697 (local address 127.30.33.1:57277)
0810 23:43:51.051967 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.051986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:51.052125 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:51.052437 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:51.052439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:51.052460 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:51.052753 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:51.052760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.053920 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.053925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:51.056095 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:51.056107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.057633 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.057639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:51.057834 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.057861 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:43:51.057937 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:43:51.058074 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":27,"thread_start_us":305,"threads_started":1}
I20260810 23:43:51.058465 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.050753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:38675 (local address 127.30.33.2:40669)
0810 23:43:51.050980 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:51.050986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:51.051285 (+ 299us) server_negotiation.cc:407] Connection header received
0810 23:43:51.051566 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:51.051574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:51.051675 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:51.051776 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:51.052389 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.053321 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.056121 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.057301 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.057445 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.057547 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:51.058015 (+ 468us) server_negotiation.cc:299] Negotiation successful
0810 23:43:51.058151 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:43:51.058727 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.051502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.33.1:57277 (local address 127.30.33.3:43697)
0810 23:43:51.051713 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:51.051718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:51.052075 (+ 357us) server_negotiation.cc:407] Connection header received
0810 23:43:51.052255 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:51.052259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:51.052322 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:51.052417 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:43:51.052898 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.053791 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.056281 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.057455 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.057569 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.057671 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:51.058001 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:43:51.058190 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:43:51.059139 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.050529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.2:40669 (local address 127.30.33.1:38675)
0810 23:43:51.051166 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.051182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:43:51.051348 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:43:51.051811 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:43:51.051816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:43:51.051843 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:43:51.052248 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:51.052261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.053471 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.053476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:51.055901 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:43:51.055920 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.057491 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.057495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:43:51.057723 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.057798 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 23:43:51.058721 (+ 923us) client_negotiation.cc:240] Negotiation successful
0810 23:43:51.058854 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":126,"thread_start_us":326,"threads_started":1}
I20260810 23:43:51.060146 4129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3a5e2342214402a455bb95c6dc3c87" candidate_uuid: "3724716d66b6466ab9ccadb198e9dd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" is_pre_election: true
I20260810 23:43:51.060209 3888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3a5e2342214402a455bb95c6dc3c87" candidate_uuid: "3724716d66b6466ab9ccadb198e9dd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1dbde6deec74b4ebc3060bc1093631b" is_pre_election: true
I20260810 23:43:51.060817 4129 raft_consensus.cc:2454] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3724716d66b6466ab9ccadb198e9dd38 in term 0.
I20260810 23:43:51.060824 3888 raft_consensus.cc:2454] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3724716d66b6466ab9ccadb198e9dd38 in term 0.
I20260810 23:43:51.061595 3541 leader_election.cc:302] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3724716d66b6466ab9ccadb198e9dd38, e3b30e06dba14718b0fb5d4aacf6d81a; no voters:
I20260810 23:43:51.062011 4268 raft_consensus.cc:2781] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:43:51.062173 4268 raft_consensus.cc:482] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:43:51.062300 4268 raft_consensus.cc:3037] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:51.064639 4268 raft_consensus.cc:504] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:51.065371 4268 leader_election.cc:288] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [CANDIDATE]: Term 1 election: Requested vote from peers d1dbde6deec74b4ebc3060bc1093631b (127.30.33.2:40669), e3b30e06dba14718b0fb5d4aacf6d81a (127.30.33.3:43697)
I20260810 23:43:51.066040 3888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3a5e2342214402a455bb95c6dc3c87" candidate_uuid: "3724716d66b6466ab9ccadb198e9dd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1dbde6deec74b4ebc3060bc1093631b"
I20260810 23:43:51.066085 4129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3a5e2342214402a455bb95c6dc3c87" candidate_uuid: "3724716d66b6466ab9ccadb198e9dd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a"
I20260810 23:43:51.066332 4129 raft_consensus.cc:3037] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:51.066332 3888 raft_consensus.cc:3037] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:51.069674 3888 raft_consensus.cc:2454] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3724716d66b6466ab9ccadb198e9dd38 in term 1.
I20260810 23:43:51.069681 4129 raft_consensus.cc:2454] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3724716d66b6466ab9ccadb198e9dd38 in term 1.
I20260810 23:43:51.070230 3541 leader_election.cc:302] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3724716d66b6466ab9ccadb198e9dd38, d1dbde6deec74b4ebc3060bc1093631b; no voters:
I20260810 23:43:51.070582 4268 raft_consensus.cc:2781] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:51.071072 4268 raft_consensus.cc:686] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 1 LEADER]: Becoming Leader. State: Replica: 3724716d66b6466ab9ccadb198e9dd38, State: Running, Role: LEADER
I20260810 23:43:51.071686 4268 consensus_queue.cc:237] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } }
I20260810 23:43:51.077057 3413 catalog_manager.cc:5495] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3724716d66b6466ab9ccadb198e9dd38 (127.30.33.1). New cstate: current_term: 1 leader_uuid: "3724716d66b6466ab9ccadb198e9dd38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3724716d66b6466ab9ccadb198e9dd38" member_type: VOTER last_known_addr { host: "127.30.33.1" port: 43817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 } health_report { overall_health: UNKNOWN } } }
I20260810 23:43:51.096882 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.090283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.30.33.1:43817)
0810 23:43:51.090497 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:51.090503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:51.090523 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:43:51.090639 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:51.090645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:51.090781 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:51.090937 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:43:51.091533 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.092254 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.094370 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.094858 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.095052 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.095682 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:43:51.096266 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:43:51.096382 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:51.096556 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:43:51.096636 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:43:51.105711 3888 raft_consensus.cc:1261] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 1 FOLLOWER]: Refusing update from remote peer 3724716d66b6466ab9ccadb198e9dd38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:51.105712 4129 raft_consensus.cc:1261] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 1 FOLLOWER]: Refusing update from remote peer 3724716d66b6466ab9ccadb198e9dd38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:43:51.106804 4268 consensus_queue.cc:1035] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1dbde6deec74b4ebc3060bc1093631b" member_type: VOTER last_known_addr { host: "127.30.33.2" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:51.107249 4272 consensus_queue.cc:1035] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b30e06dba14718b0fb5d4aacf6d81a" member_type: VOTER last_known_addr { host: "127.30.33.3" port: 43697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:43:51.136782 3411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:51.137149 3411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:43:51.141956 3411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b17ff1e88e58431c98e47f4640e04e29: Sending DeleteTablet for 3 replicas of tablet ec3a5e2342214402a455bb95c6dc3c87
I20260810 23:43:51.143394 3626 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3a5e2342214402a455bb95c6dc3c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:51 UTC) from {username='slave'} at 127.0.0.1:34400
I20260810 23:43:51.143572 3852 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3a5e2342214402a455bb95c6dc3c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:51 UTC) from {username='slave'} at 127.0.0.1:57970
I20260810 23:43:51.143718 4100 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3a5e2342214402a455bb95c6dc3c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:43:51 UTC) from {username='slave'} at 127.0.0.1:35356
I20260810 23:43:51.145715 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3511
I20260810 23:43:51.154983 4297 tablet_replica.cc:330] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: stopping tablet replica
I20260810 23:43:51.155571 4297 raft_consensus.cc:2229] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:51.156180 4297 raft_consensus.cc:2258] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:51.161576 4299 tablet_replica.cc:330] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: stopping tablet replica
I20260810 23:43:51.162788 4299 raft_consensus.cc:2229] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:43:51.163808 4297 ts_tablet_manager.cc:1905] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:51.164479 4299 raft_consensus.cc:2258] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:51.167563 4299 ts_tablet_manager.cc:1905] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:43:51.171125 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.147219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.30.33.1:43817)
0810 23:43:51.147841 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:51.147848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:51.161222 (+ 13374us) server_negotiation.cc:407] Connection header received
0810 23:43:51.161878 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:51.161885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:51.162005 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:51.162090 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:51.163044 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.164294 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.166810 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.167505 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.167614 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.169225 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:51.169238 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:51.169257 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:51.170125 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:51.170202 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:51.170230 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:51.170331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:51.170587 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 23:43:51.170712 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467}
I20260810 23:43:51.171257 4297 ts_tablet_manager.cc:1918] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:51.171438 4297 log.cc:1196] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-1/wal/wals/ec3a5e2342214402a455bb95c6dc3c87
I20260810 23:43:51.172039 4296 tablet_replica.cc:330] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38: stopping tablet replica
I20260810 23:43:51.172205 4297 ts_tablet_manager.cc:1939] T ec3a5e2342214402a455bb95c6dc3c87 P d1dbde6deec74b4ebc3060bc1093631b: Deleting consensus metadata
I20260810 23:43:51.172531 4296 raft_consensus.cc:2229] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:43:51.173022 3727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:51.174515 3395 catalog_manager.cc:4841] TS d1dbde6deec74b4ebc3060bc1093631b (127.30.33.2:40669): tablet ec3a5e2342214402a455bb95c6dc3c87 (table test-table [id=4711887444e44ea694f1df7c317b2f7c]) successfully deleted
I20260810 23:43:51.176945 4299 ts_tablet_manager.cc:1918] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:43:51.177160 4299 log.cc:1196] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Deleting WAL directory at /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786405091056293-30852-0/minicluster-data/ts-2/wal/wals/ec3a5e2342214402a455bb95c6dc3c87
I20260810 23:43:51.177752 4299 ts_tablet_manager.cc:1939] T ec3a5e2342214402a455bb95c6dc3c87 P e3b30e06dba14718b0fb5d4aacf6d81a: Deleting consensus metadata
I20260810 23:43:51.179617 3395 catalog_manager.cc:4841] TS e3b30e06dba14718b0fb5d4aacf6d81a (127.30.33.3:43697): tablet ec3a5e2342214402a455bb95c6dc3c87 (table test-table [id=4711887444e44ea694f1df7c317b2f7c]) successfully deleted
I20260810 23:43:51.273974 4296 raft_consensus.cc:2258] T ec3a5e2342214402a455bb95c6dc3c87 P 3724716d66b6466ab9ccadb198e9dd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:43:51.274637 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3511
W20260810 23:43:51.297372 3390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.33.1:43817 (error 108)
I20260810 23:43:51.297673 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3746
I20260810 23:43:51.299333 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.298609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34432)
0810 23:43:51.299053 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.299169 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":232,"threads_started":1}
W20260810 23:43:51.299577 3390 catalog_manager.cc:4584] TS 3724716d66b6466ab9ccadb198e9dd38 (127.30.33.1:43817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec3a5e2342214402a455bb95c6dc3c87: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
I20260810 23:43:51.304915 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.298667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.30.33.2:40669)
0810 23:43:51.298858 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:51.298864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:51.298886 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:43:51.298992 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:51.298998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:51.299107 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:51.299221 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:51.299680 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.300470 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.301760 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.302198 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.302316 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.303409 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:51.303418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:51.303434 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:51.304109 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:51.304165 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:51.304196 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:51.304312 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:51.304448 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:43:51.304564 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:43:51.305645 3967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:51.353209 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.352624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34444)
0810 23:43:51.352877 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.353008 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 23:43:51.354462 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.353991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34450)
0810 23:43:51.354177 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.354272 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:43:51.401722 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3746
I20260810 23:43:51.414872 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.414406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34464)
0810 23:43:51.414620 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.414710 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:43:51.416002 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.415627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34472)
0810 23:43:51.415786 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.415863 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:43:51.433393 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3979
I20260810 23:43:51.441330 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.434197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.30.33.3:43697)
0810 23:43:51.434384 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:51.434391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:51.434490 (+ 99us) server_negotiation.cc:407] Connection header received
0810 23:43:51.434704 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:51.434710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:51.434806 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:51.434905 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:51.435335 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.436217 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.437489 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:51.437936 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:51.438049 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:51.439201 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:51.439210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:51.439226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:51.440057 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:51.440131 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:51.440154 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:51.440264 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:51.440505 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:43:51.440709 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:43:51.442215 4201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:51.496999 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.496325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34484)
0810 23:43:51.496643 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.496778 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 23:43:51.498332 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:51.497804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.1:43817 (local address 127.0.0.1:34492)
0810 23:43:51.498049 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 23:43:51.498147 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.1:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 23:43:51.530988 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3979
I20260810 23:43:51.556593 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 3351
I20260810 23:43:51.557722 3463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:51.669858 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 3351
2026-08-10T23:43:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3148 ms)
[----------] 6 tests from EncryptionPolicyTest (24319 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:43:51 dist-test-slave-pww0 krb5kdc[4312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:43:51 dist-test-slave-pww0 krb5kdc[4312](info): set up 2 sockets
Aug 10 23:43:51 dist-test-slave-pww0 krb5kdc[4312](info): commencing operation
krb5kdc: starting...
W20260810 23:43:52.998688 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:43:53 dist-test-slave-pww0 krb5kdc[4312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:43:53.351303 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:53.710031 4328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:53.710388 4328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:53.710493 4328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:53.717940 4328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:53.718075 4328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:53.718133 4328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:53.718196 4328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:53.718240 4328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:53.728601 4328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:53.730001 4328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:53.731364 4328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:53.743535 4345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:53.744340 4347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:53.744403 4343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:53.744302 4328 server_base.cc:1034] running on GCE node
I20260810 23:43:53.745466 4328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:53.747097 4328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:53.748338 4328 hybrid_clock.cc:648] HybridClock initialized: now 1786405433748281 us; error 48 us; skew 500 ppm
I20260810 23:43:53.748729 4328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:53 dist-test-slave-pww0 krb5kdc[4312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:53.756585 4328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:43:53.758266 4328 webserver.cc:460] Webserver started at http://127.0.0.1:34797/ using document root <none> and password file <none>
I20260810 23:43:53.759001 4328 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:53.759141 4328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:53.759445 4328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:53.762748 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "9b1ef4e40fd4476da7c9bb3469898772"
format_stamp: "Formatted at 2026-08-10 23:43:53 on dist-test-slave-pww0"
server_key: "dfe55b589388e9f0150ad07e11d6c20c"
server_key_iv: "850b5b94ef1b47d8eca27b414180db92"
server_key_version: "encryptionkey@0"
I20260810 23:43:53.763700 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "9b1ef4e40fd4476da7c9bb3469898772"
format_stamp: "Formatted at 2026-08-10 23:43:53 on dist-test-slave-pww0"
server_key: "dfe55b589388e9f0150ad07e11d6c20c"
server_key_iv: "850b5b94ef1b47d8eca27b414180db92"
server_key_version: "encryptionkey@0"
I20260810 23:43:53.769660 4328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 23:43:53.773829 4361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:53.775391 4328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:43:53.775616 4328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "9b1ef4e40fd4476da7c9bb3469898772"
format_stamp: "Formatted at 2026-08-10 23:43:53 on dist-test-slave-pww0"
server_key: "dfe55b589388e9f0150ad07e11d6c20c"
server_key_iv: "850b5b94ef1b47d8eca27b414180db92"
server_key_version: "encryptionkey@0"
I20260810 23:43:53.775826 4328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:53.798380 4328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:53.803076 4328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:53.803382 4328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:53.821728 4328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42699
I20260810 23:43:53.821779 4451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42699 every 8 connection(s)
I20260810 23:43:53.823149 4328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:53.826471 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4328
I20260810 23:43:53.826670 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:53.827137 4453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:53.827338 30852 external_mini_cluster.cc:1428] Setting key f5cf7172b9a2c3da3f20fa543bfce826
I20260810 23:43:53.842459 4453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772: Bootstrap starting.
Aug 10 23:43:53 dist-test-slave-pww0 krb5kdc[4312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:43:53.846480 4453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:53.847760 4453 log.cc:824] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:53.851086 4453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772: No bootstrap required, opened a new log
I20260810 23:43:53.853175 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:53.829342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60267 (local address 127.0.0.1:42699)
0810 23:43:53.830184 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:53.830200 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:53.830261 (+ 61us) server_negotiation.cc:407] Connection header received
0810 23:43:53.832143 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:53.832167 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:53.832636 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:53.833004 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:53.835016 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:53.836484 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:53.838385 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:53.839234 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:53.839394 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:53.844927 (+ 5533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:53.844939 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:53.844957 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:53.845012 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:53.848341 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:53.849330 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:53.849338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:53.849345 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:53.849523 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:53.850104 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:53.850108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:53.850110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:53.850721 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:53.850988 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:53.851635 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 23:43:53.852014 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":369,"threads_started":1}
I20260810 23:43:53.857658 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:43:53.860781 4453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ef4e40fd4476da7c9bb3469898772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260810 23:43:53.861359 4453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:53.861492 4453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1ef4e40fd4476da7c9bb3469898772, State: Initialized, Role: FOLLOWER
I20260810 23:43:53.862282 4453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ef4e40fd4476da7c9bb3469898772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260810 23:43:53.862634 4453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:53.862788 4453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:53.862995 4453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:53.866349 4453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ef4e40fd4476da7c9bb3469898772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260810 23:43:53.867064 4453 leader_election.cc:302] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b1ef4e40fd4476da7c9bb3469898772; no voters:
I20260810 23:43:53.867861 4453 leader_election.cc:288] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:53.867974 4467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:53.868641 4467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [term 1 LEADER]: Becoming Leader. State: Replica: 9b1ef4e40fd4476da7c9bb3469898772, State: Running, Role: LEADER
I20260810 23:43:53.869311 4467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ef4e40fd4476da7c9bb3469898772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } }
I20260810 23:43:53.873677 4453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:53.874358 4469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b1ef4e40fd4476da7c9bb3469898772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ef4e40fd4476da7c9bb3469898772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } } }
I20260810 23:43:53.874790 4469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:53.878691 4473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:53.879055 4468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b1ef4e40fd4476da7c9bb3469898772. Latest consensus state: current_term: 1 leader_uuid: "9b1ef4e40fd4476da7c9bb3469898772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ef4e40fd4476da7c9bb3469898772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42699 } } }
I20260810 23:43:53.879935 4468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:53.882555 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:53.864194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51331 (local address 127.0.0.1:42699)
0810 23:43:53.864418 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:53.864425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:53.864807 (+ 382us) server_negotiation.cc:407] Connection header received
0810 23:43:53.865011 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:53.865015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:53.865088 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:53.865174 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:53.867160 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:53.868094 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:53.869685 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:53.870254 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:53.870376 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:53.873208 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:53.873217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:53.873222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:43:53.873274 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:53.877387 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:53.878610 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:53.878614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:53.878616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:53.878753 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:43:53.879365 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:43:53.879370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:43:53.879374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:43:53.879990 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:53.880336 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:53.881825 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0810 23:43:53.882057 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:43:53.886435 4473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:53.896744 4473 catalog_manager.cc:1347] Generated new cluster ID: 4f5d93ec46184004a1db8bbcafececbd
I20260810 23:43:53.896878 4473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:53.914462 4473 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:53.915907 4473 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:53.923409 4473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b1ef4e40fd4476da7c9bb3469898772: Generated new TSK 0
I20260810 23:43:53.924335 4473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:43:53.934863 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4328
I20260810 23:43:53.936798 4446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:43:54.029585 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4328
W20260810 23:43:54.053063 4309 connection.cc:535] client connection to 127.0.0.1:42699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42699): Connection reset by peer (error 104)
2026-08-10T23:43:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:43:54 dist-test-slave-pww0 krb5kdc[4514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:43:54 dist-test-slave-pww0 krb5kdc[4514](info): set up 2 sockets
Aug 10 23:43:54 dist-test-slave-pww0 krb5kdc[4514](info): commencing operation
krb5kdc: starting...
W20260810 23:43:55.372599 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:43:55 dist-test-slave-pww0 krb5kdc[4514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:43:55.719074 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:56.076982 4538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:56.077315 4538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:56.077430 4538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:56.084933 4538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:56.085074 4538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:56.085130 4538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:56.085193 4538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:56.085239 4538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:56.095664 4538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:56.097064 4538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:56.098412 4538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:56.111522 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:56.111429 4551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:56.111673 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:56.112857 4538 server_base.cc:1034] running on GCE node
I20260810 23:43:56.113722 4538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:56.115298 4538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:56.116513 4538 hybrid_clock.cc:648] HybridClock initialized: now 1786405436116451 us; error 55 us; skew 500 ppm
I20260810 23:43:56.116880 4538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:56 dist-test-slave-pww0 krb5kdc[4514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:56.120230 4538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:43:56.121766 4538 webserver.cc:460] Webserver started at http://127.0.0.1:38921/ using document root <none> and password file <none>
I20260810 23:43:56.122493 4538 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:56.122625 4538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:56.122961 4538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:56.126222 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "061bcdfb5f4547a78b78855732ac7d4a"
format_stamp: "Formatted at 2026-08-10 23:43:56 on dist-test-slave-pww0"
server_key: "0bcd58ef3b516d3a8a43b70dc7903374"
server_key_iv: "983639d1578bd5a3d8ff32841c6f87cb"
server_key_version: "encryptionkey@0"
I20260810 23:43:56.127018 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "061bcdfb5f4547a78b78855732ac7d4a"
format_stamp: "Formatted at 2026-08-10 23:43:56 on dist-test-slave-pww0"
server_key: "0bcd58ef3b516d3a8a43b70dc7903374"
server_key_iv: "983639d1578bd5a3d8ff32841c6f87cb"
server_key_version: "encryptionkey@0"
I20260810 23:43:56.132572 4538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:43:56.136447 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:56.138082 4538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:56.138295 4538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "061bcdfb5f4547a78b78855732ac7d4a"
format_stamp: "Formatted at 2026-08-10 23:43:56 on dist-test-slave-pww0"
server_key: "0bcd58ef3b516d3a8a43b70dc7903374"
server_key_iv: "983639d1578bd5a3d8ff32841c6f87cb"
server_key_version: "encryptionkey@0"
I20260810 23:43:56.138518 4538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:56.158856 4538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:56.163568 4538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:56.163883 4538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:56.182679 4538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42511
I20260810 23:43:56.182703 4657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42511 every 8 connection(s)
I20260810 23:43:56.184341 4538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:56.188344 4659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:56.194504 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4538
I20260810 23:43:56.194728 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:56.195315 30852 external_mini_cluster.cc:1428] Setting key 21e772c5117b4710a0699d27edba195e
I20260810 23:43:56.200354 4659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a: Bootstrap starting.
I20260810 23:43:56.203703 4659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:56.204815 4659 log.cc:824] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:56.208211 4659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a: No bootstrap required, opened a new log
I20260810 23:43:56.209169 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:56.197494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58323 (local address 127.0.0.1:42511)
0810 23:43:56.198412 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:56.198425 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:56.198472 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:43:56.200281 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:56.200307 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:56.200777 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:56.201156 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:56.201751 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:56.203049 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:56.204503 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:56.205253 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:56.205457 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:56.206760 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:56.206769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:56.206784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:56.207355 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:56.207435 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:56.207485 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:56.207630 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:56.207917 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 23:43:56.208113 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":414,"threads_started":1}
I20260810 23:43:56.213433 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:43:56.217823 4659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061bcdfb5f4547a78b78855732ac7d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260810 23:43:56.218402 4659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:56.218537 4659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061bcdfb5f4547a78b78855732ac7d4a, State: Initialized, Role: FOLLOWER
I20260810 23:43:56.219283 4659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061bcdfb5f4547a78b78855732ac7d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260810 23:43:56.219587 4659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:56.219743 4659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:56.219961 4659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:56.223084 4659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061bcdfb5f4547a78b78855732ac7d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260810 23:43:56.223809 4659 leader_election.cc:302] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 061bcdfb5f4547a78b78855732ac7d4a; no voters:
I20260810 23:43:56.224502 4659 leader_election.cc:288] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:56.224606 4682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:56.225728 4682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [term 1 LEADER]: Becoming Leader. State: Replica: 061bcdfb5f4547a78b78855732ac7d4a, State: Running, Role: LEADER
I20260810 23:43:56.226490 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:56.218976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43885 (local address 127.0.0.1:42511)
0810 23:43:56.219169 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:56.219175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:56.219714 (+ 539us) server_negotiation.cc:407] Connection header received
0810 23:43:56.219935 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:56.219941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:56.220016 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:56.220114 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:56.220801 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:56.221638 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:56.223537 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:43:56.224063 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:43:56.224173 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:43:56.225520 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:56.225528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:56.225533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:56.225565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:56.225639 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:56.225665 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:56.225800 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:56.226090 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 23:43:56.226152 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:43:56.226492 4682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061bcdfb5f4547a78b78855732ac7d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260810 23:43:56.227174 4659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:56.229385 4685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061bcdfb5f4547a78b78855732ac7d4a. Latest consensus state: current_term: 1 leader_uuid: "061bcdfb5f4547a78b78855732ac7d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061bcdfb5f4547a78b78855732ac7d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } } }
I20260810 23:43:56.229813 4685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:56.230396 4684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "061bcdfb5f4547a78b78855732ac7d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061bcdfb5f4547a78b78855732ac7d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } } }
I20260810 23:43:56.230777 4684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:56.231729 4688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:56.238138 4688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:56.249387 4688 catalog_manager.cc:1347] Generated new cluster ID: 85ce67ca46fd447db884eb7ab635d23a
I20260810 23:43:56.249522 4688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:56.262096 4688 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:56.263501 4688 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:56.275004 4688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 061bcdfb5f4547a78b78855732ac7d4a: Generated new TSK 0
I20260810 23:43:56.276139 4688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:43:56.294246 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4538
W20260810 23:43:56.295514 4650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58323
W20260810 23:43:56.296401 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:43:56.296590 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4538
W20260810 23:43:56.334303 4512 connection.cc:535] client connection to 127.0.0.1:42511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42511): Connection reset by peer (error 104)
2026-08-10T23:43:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:43:56 dist-test-slave-pww0 krb5kdc[4724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:43:56 dist-test-slave-pww0 krb5kdc[4724](info): set up 2 sockets
Aug 10 23:43:56 dist-test-slave-pww0 krb5kdc[4724](info): commencing operation
krb5kdc: starting...
W20260810 23:43:57.623260 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:43:57 dist-test-slave-pww0 krb5kdc[4724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:43:57.975641 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:43:58.338662 4742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:43:58.339004 4742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:43:58.339141 4742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:43:58.346706 4742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:43:58.346859 4742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:43:58.346910 4742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:43:58.346995 4742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:43:58.347061 4742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:43:58.357709 4742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:43:58.359105 4742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:43:58.360539 4742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:43:58.373488 4757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:58.373556 4750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:43:58.373534 4752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:43:58.374495 4742 server_base.cc:1034] running on GCE node
I20260810 23:43:58.375427 4742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:43:58.377130 4742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:43:58.378369 4742 hybrid_clock.cc:648] HybridClock initialized: now 1786405438378320 us; error 50 us; skew 500 ppm
I20260810 23:43:58.378748 4742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:43:58 dist-test-slave-pww0 krb5kdc[4724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:43:58.382196 4742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:43:58.384008 4742 webserver.cc:460] Webserver started at http://127.0.0.1:34423/ using document root <none> and password file <none>
I20260810 23:43:58.384761 4742 fs_manager.cc:359] Metadata directory not provided
I20260810 23:43:58.384887 4742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:43:58.385183 4742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:43:58.388706 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "e18b7075f4df4dae85884723293733f4"
format_stamp: "Formatted at 2026-08-10 23:43:58 on dist-test-slave-pww0"
server_key: "73f4dac1c316df03bc92ad1a7f27d206"
server_key_iv: "eeda0f67da95fd7b4a3e051c9df56aab"
server_key_version: "encryptionkey@0"
I20260810 23:43:58.389565 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "e18b7075f4df4dae85884723293733f4"
format_stamp: "Formatted at 2026-08-10 23:43:58 on dist-test-slave-pww0"
server_key: "73f4dac1c316df03bc92ad1a7f27d206"
server_key_iv: "eeda0f67da95fd7b4a3e051c9df56aab"
server_key_version: "encryptionkey@0"
I20260810 23:43:58.395691 4742 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 23:43:58.399875 4766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:43:58.401449 4742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:43:58.401639 4742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "e18b7075f4df4dae85884723293733f4"
format_stamp: "Formatted at 2026-08-10 23:43:58 on dist-test-slave-pww0"
server_key: "73f4dac1c316df03bc92ad1a7f27d206"
server_key_iv: "eeda0f67da95fd7b4a3e051c9df56aab"
server_key_version: "encryptionkey@0"
I20260810 23:43:58.401858 4742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:43:58.408497 4742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:43:58.413323 4742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:43:58.413645 4742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:43:58.432583 4742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46059
I20260810 23:43:58.432627 4869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46059 every 8 connection(s)
I20260810 23:43:58.434150 4742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:43:58.438275 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:43:58.441664 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4742
I20260810 23:43:58.441877 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:43:58.442408 30852 external_mini_cluster.cc:1428] Setting key 59def0ebe93cf52996b88730550df82c
I20260810 23:43:58.450618 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:58.444436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55185 (local address 127.0.0.1:46059)
0810 23:43:58.445253 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:58.445266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:58.445313 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:43:58.447041 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:58.447067 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:58.447527 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:58.447930 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:58.449023 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:58.449030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:58.449043 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:58.449057 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:58.449118 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:58.449139 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:58.449327 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:58.449427 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:43:58.449604 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":349,"threads_started":1}
I20260810 23:43:58.451184 4870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4: Bootstrap starting.
I20260810 23:43:58.454411 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:43:58.454660 4870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4: Neither blocks nor log segments found. Creating new log.
I20260810 23:43:58.455829 4870 log.cc:824] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:43:58.459391 4870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4: No bootstrap required, opened a new log
I20260810 23:43:58.461726 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:43:58.459084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42173 (local address 127.0.0.1:46059)
0810 23:43:58.459262 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:43:58.459268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:43:58.459700 (+ 432us) server_negotiation.cc:407] Connection header received
0810 23:43:58.459891 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:43:58.459895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:43:58.459947 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:43:58.460035 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:43:58.460982 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:43:58.460988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:43:58.460996 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:43:58.461056 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:43:58.461102 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:43:58.461123 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:43:58.461270 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:43:58.461380 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:43:58.461449 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:43:58.468019 4870 raft_consensus.cc:348] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b7075f4df4dae85884723293733f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260810 23:43:58.468587 4870 raft_consensus.cc:374] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:43:58.468711 4870 raft_consensus.cc:729] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18b7075f4df4dae85884723293733f4, State: Initialized, Role: FOLLOWER
I20260810 23:43:58.469408 4870 consensus_queue.cc:260] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b7075f4df4dae85884723293733f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260810 23:43:58.469648 4870 raft_consensus.cc:388] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:43:58.469800 4870 raft_consensus.cc:482] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:43:58.469960 4870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:43:58.472544 4870 raft_consensus.cc:504] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b7075f4df4dae85884723293733f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260810 23:43:58.473165 4870 leader_election.cc:302] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e18b7075f4df4dae85884723293733f4; no voters:
I20260810 23:43:58.473800 4870 leader_election.cc:288] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:43:58.473937 4891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:43:58.474575 4891 raft_consensus.cc:686] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [term 1 LEADER]: Becoming Leader. State: Replica: e18b7075f4df4dae85884723293733f4, State: Running, Role: LEADER
I20260810 23:43:58.475116 4891 consensus_queue.cc:237] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b7075f4df4dae85884723293733f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260810 23:43:58.475899 4870 sys_catalog.cc:564] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:43:58.478106 4893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e18b7075f4df4dae85884723293733f4. Latest consensus state: current_term: 1 leader_uuid: "e18b7075f4df4dae85884723293733f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b7075f4df4dae85884723293733f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } } }
I20260810 23:43:58.478186 4892 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e18b7075f4df4dae85884723293733f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18b7075f4df4dae85884723293733f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } } }
I20260810 23:43:58.478638 4893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:58.478644 4892 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:43:58.481349 4903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:43:58.487180 4903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:43:58.496652 4903 catalog_manager.cc:1347] Generated new cluster ID: 7a4ef62c21064614bfa65069a6987c61
I20260810 23:43:58.496783 4903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:43:58.513185 4903 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:43:58.513759 4903 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:43:58.524277 4903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e18b7075f4df4dae85884723293733f4: Generated new TSK 0
I20260810 23:43:58.525607 4903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:43:58.537227 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4742
W20260810 23:43:58.538547 4862 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55185
W20260810 23:43:58.539530 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4742: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:43:58.539727 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4742
2026-08-10T23:43:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:43:58 dist-test-slave-pww0 krb5kdc[4930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:43:58 dist-test-slave-pww0 krb5kdc[4930](info): set up 2 sockets
Aug 10 23:43:58 dist-test-slave-pww0 krb5kdc[4930](info): commencing operation
krb5kdc: starting...
W20260810 23:43:59.857759 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:00 dist-test-slave-pww0 krb5kdc[4930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:00.220755 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:00.579174 4958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:00.579555 4958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:00.579671 4958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:00.587210 4958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:00.587348 4958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:00.587399 4958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:00.587484 4958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:00.587548 4958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:00.597965 4958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:00.599335 4958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:00.600752 4958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:00.613013 4975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:00.613037 4978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:00.613101 4976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:00.614097 4958 server_base.cc:1034] running on GCE node
I20260810 23:44:00.615010 4958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:00.616710 4958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:00.617966 4958 hybrid_clock.cc:648] HybridClock initialized: now 1786405440617912 us; error 58 us; skew 500 ppm
I20260810 23:44:00.618363 4958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:00 dist-test-slave-pww0 krb5kdc[4930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:00.621783 4958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:00.623538 4958 webserver.cc:460] Webserver started at http://127.0.0.1:43449/ using document root <none> and password file <none>
I20260810 23:44:00.624260 4958 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:00.624377 4958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:00.624646 4958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:00.627902 4958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "65472d76acc541e585863ede969556a6"
format_stamp: "Formatted at 2026-08-10 23:44:00 on dist-test-slave-pww0"
server_key: "68e8df724b1f37fc7dd0671810527bba"
server_key_iv: "214eeed9412b1973b771b9f160f38449"
server_key_version: "encryptionkey@0"
I20260810 23:44:00.628727 4958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "65472d76acc541e585863ede969556a6"
format_stamp: "Formatted at 2026-08-10 23:44:00 on dist-test-slave-pww0"
server_key: "68e8df724b1f37fc7dd0671810527bba"
server_key_iv: "214eeed9412b1973b771b9f160f38449"
server_key_version: "encryptionkey@0"
I20260810 23:44:00.634435 4958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:44:00.638537 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:00.640260 4958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:44:00.640475 4958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "65472d76acc541e585863ede969556a6"
format_stamp: "Formatted at 2026-08-10 23:44:00 on dist-test-slave-pww0"
server_key: "68e8df724b1f37fc7dd0671810527bba"
server_key_iv: "214eeed9412b1973b771b9f160f38449"
server_key_version: "encryptionkey@0"
I20260810 23:44:00.640720 4958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:00.664573 4958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:00.669335 4958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:00.669649 4958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:00.689045 4958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35401
I20260810 23:44:00.689066 5078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35401 every 8 connection(s)
I20260810 23:44:00.690683 4958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:00.694842 5080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:00.696534 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 4958
I20260810 23:44:00.696717 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:00.697232 30852 external_mini_cluster.cc:1428] Setting key 42c2f55861351dd657fa4d323a785190
I20260810 23:44:00.710520 5080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6: Bootstrap starting.
Aug 10 23:44:00 dist-test-slave-pww0 krb5kdc[4930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:44:00.714866 5080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:00.716459 5080 log.cc:824] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:00.720642 5080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6: No bootstrap required, opened a new log
I20260810 23:44:00.725394 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:00.699304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53797 (local address 127.0.0.1:35401)
0810 23:44:00.700279 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:00.700299 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:00.700365 (+ 66us) server_negotiation.cc:407] Connection header received
0810 23:44:00.702135 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:00.702174 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:00.702804 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:00.703289 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:00.704814 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:00.706877 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:00.708669 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:00.709783 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:00.710002 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:00.715422 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:00.715438 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:00.715494 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:00.715563 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:00.719887 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:00.720942 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:00.720952 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:00.720960 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:00.721207 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:00.721858 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:00.721864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:00.721866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:00.722638 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:00.722953 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:00.723625 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 23:44:00.724151 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":469,"threads_started":1}
I20260810 23:44:00.729916 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:00.729776 5080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65472d76acc541e585863ede969556a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260810 23:44:00.730329 5080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:00.730444 5080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65472d76acc541e585863ede969556a6, State: Initialized, Role: FOLLOWER
I20260810 23:44:00.731166 5080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65472d76acc541e585863ede969556a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260810 23:44:00.731525 5080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:00.731801 5080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:00.732013 5080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:00.735383 5080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65472d76acc541e585863ede969556a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260810 23:44:00.736137 5080 leader_election.cc:302] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65472d76acc541e585863ede969556a6; no voters:
I20260810 23:44:00.736923 5080 leader_election.cc:288] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:00.737018 5097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:00.738988 5097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [term 1 LEADER]: Becoming Leader. State: Replica: 65472d76acc541e585863ede969556a6, State: Running, Role: LEADER
I20260810 23:44:00.739584 5080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:00.739785 5097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65472d76acc541e585863ede969556a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260810 23:44:00.743647 5098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65472d76acc541e585863ede969556a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65472d76acc541e585863ede969556a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } } }
I20260810 23:44:00.743851 5099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65472d76acc541e585863ede969556a6. Latest consensus state: current_term: 1 leader_uuid: "65472d76acc541e585863ede969556a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65472d76acc541e585863ede969556a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } } }
I20260810 23:44:00.744113 5099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:00.744113 5098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:00.745312 5116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:00.754101 5116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:00.757262 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:00.735485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52703 (local address 127.0.0.1:35401)
0810 23:44:00.735688 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:00.735696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:00.736264 (+ 568us) server_negotiation.cc:407] Connection header received
0810 23:44:00.736469 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:00.736475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:00.736546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:00.736647 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:00.738785 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:00.739988 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:00.741925 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:00.742845 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:00.742981 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:00.747281 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:00.747290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:00.747296 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:00.747350 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:00.751977 (+ 4627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:00.753241 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:00.753249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:00.753254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:00.753463 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:00.754023 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:00.754029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:00.754033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:00.754649 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:00.754970 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:00.756481 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0810 23:44:00.756797 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:44:00.766692 5116 catalog_manager.cc:1347] Generated new cluster ID: ebcf1cb0639a4e959338afc7dada8087
I20260810 23:44:00.766832 5116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:00.809098 5116 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:00.811051 5116 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:00.819263 5116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65472d76acc541e585863ede969556a6: Generated new TSK 0
I20260810 23:44:00.820441 5116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:00.829066 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 4958
I20260810 23:44:00.830664 5070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:00.926311 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 4958
2026-08-10T23:44:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:01 dist-test-slave-pww0 krb5kdc[5140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:01 dist-test-slave-pww0 krb5kdc[5140](info): set up 2 sockets
Aug 10 23:44:01 dist-test-slave-pww0 krb5kdc[5140](info): commencing operation
krb5kdc: starting...
W20260810 23:44:02.236799 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:02 dist-test-slave-pww0 krb5kdc[5140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:02.595583 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:02.954491 5156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:02.954839 5156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:02.954936 5156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:02.962555 5156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:02.962702 5156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:02.962750 5156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:02.962812 5156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:02.962857 5156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:02.973486 5156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:02.974928 5156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:02.976399 5156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:02.989585 5168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:02.989929 5166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:02.990934 5170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:02.991973 5156 server_base.cc:1034] running on GCE node
I20260810 23:44:02.992878 5156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:02.994472 5156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:02.995718 5156 hybrid_clock.cc:648] HybridClock initialized: now 1786405442995648 us; error 65 us; skew 500 ppm
I20260810 23:44:02.996146 5156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:02 dist-test-slave-pww0 krb5kdc[5140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:02.999558 5156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:03.001303 5156 webserver.cc:460] Webserver started at http://127.0.0.1:36371/ using document root <none> and password file <none>
I20260810 23:44:03.002100 5156 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:03.002241 5156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:03.002564 5156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:03.005849 5156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "d4f60ee0921a40cda021e448bea57cb9"
format_stamp: "Formatted at 2026-08-10 23:44:02 on dist-test-slave-pww0"
server_key: "7a4827b34c181f68bc54340045600ff6"
server_key_iv: "6ff6f647a16e75395d3f513bdd11ec91"
server_key_version: "encryptionkey@0"
I20260810 23:44:03.006760 5156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "d4f60ee0921a40cda021e448bea57cb9"
format_stamp: "Formatted at 2026-08-10 23:44:02 on dist-test-slave-pww0"
server_key: "7a4827b34c181f68bc54340045600ff6"
server_key_iv: "6ff6f647a16e75395d3f513bdd11ec91"
server_key_version: "encryptionkey@0"
I20260810 23:44:03.012809 5156 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260810 23:44:03.016748 5183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:03.018265 5156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:44:03.018461 5156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "d4f60ee0921a40cda021e448bea57cb9"
format_stamp: "Formatted at 2026-08-10 23:44:02 on dist-test-slave-pww0"
server_key: "7a4827b34c181f68bc54340045600ff6"
server_key_iv: "6ff6f647a16e75395d3f513bdd11ec91"
server_key_version: "encryptionkey@0"
I20260810 23:44:03.018680 5156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:03.033267 5156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:03.037766 5156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:03.038058 5156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:03.057173 5156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44941
I20260810 23:44:03.057207 5267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44941 every 8 connection(s)
I20260810 23:44:03.058734 5156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:03.061156 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5156
I20260810 23:44:03.061370 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:03.061954 30852 external_mini_cluster.cc:1428] Setting key 50620d9966323542967e1e2a6f4a25dc
I20260810 23:44:03.063787 5268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:03.076934 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:03.064113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43353 (local address 127.0.0.1:44941)
0810 23:44:03.065000 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:03.065015 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:03.065064 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:44:03.066830 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:03.066859 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:03.067353 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:03.067904 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:03.068544 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:03.069977 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:03.071494 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:03.072253 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:03.072444 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:03.073884 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:03.073897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:03.073926 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:03.074712 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:03.074966 (+ 254us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:03.075028 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:03.075171 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:03.075440 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 23:44:03.075806 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":408,"threads_started":1}
I20260810 23:44:03.077340 5268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9: Bootstrap starting.
I20260810 23:44:03.081173 5268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:03.081226 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:03.082332 5268 log.cc:824] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:03.085902 5268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9: No bootstrap required, opened a new log
I20260810 23:44:03.092440 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:03.085775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40581 (local address 127.0.0.1:44941)
0810 23:44:03.085941 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:03.085948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:03.086468 (+ 520us) server_negotiation.cc:407] Connection header received
0810 23:44:03.086679 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:03.086682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:03.086749 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:03.086843 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:03.087441 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:03.088277 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:03.089664 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:03.090167 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:03.090318 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:03.091620 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:03.091625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:03.091629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:03.091651 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:03.091704 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:03.091723 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:03.091818 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:03.091982 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 23:44:03.092086 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:44:03.094642 5268 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f60ee0921a40cda021e448bea57cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260810 23:44:03.095062 5268 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:03.095152 5268 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f60ee0921a40cda021e448bea57cb9, State: Initialized, Role: FOLLOWER
I20260810 23:44:03.095798 5268 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f60ee0921a40cda021e448bea57cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260810 23:44:03.096028 5268 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:03.096139 5268 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:03.096305 5268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:03.098839 5268 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f60ee0921a40cda021e448bea57cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260810 23:44:03.099390 5268 leader_election.cc:302] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f60ee0921a40cda021e448bea57cb9; no voters:
I20260810 23:44:03.100020 5268 leader_election.cc:288] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:03.100171 5290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:03.100967 5290 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [term 1 LEADER]: Becoming Leader. State: Replica: d4f60ee0921a40cda021e448bea57cb9, State: Running, Role: LEADER
I20260810 23:44:03.101512 5290 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f60ee0921a40cda021e448bea57cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260810 23:44:03.102234 5268 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:03.104400 5291 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f60ee0921a40cda021e448bea57cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f60ee0921a40cda021e448bea57cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } } }
I20260810 23:44:03.104734 5291 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:03.105429 5293 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f60ee0921a40cda021e448bea57cb9. Latest consensus state: current_term: 1 leader_uuid: "d4f60ee0921a40cda021e448bea57cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f60ee0921a40cda021e448bea57cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } } }
I20260810 23:44:03.105733 5293 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:03.106402 5299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:03.112129 5299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:03.121301 5299 catalog_manager.cc:1347] Generated new cluster ID: 2a2899e9cfc84d2eb24a81f1b2477050
I20260810 23:44:03.121455 5299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:03.148190 5299 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:03.150322 5299 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:03.161763 5299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f60ee0921a40cda021e448bea57cb9: Generated new TSK 0
I20260810 23:44:03.163012 5299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:03.180253 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5156
W20260810 23:44:03.182237 5260 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43353
W20260810 23:44:03.183147 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5156: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:03.183326 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5156
2026-08-10T23:44:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:03 dist-test-slave-pww0 krb5kdc[5330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:03 dist-test-slave-pww0 krb5kdc[5330](info): set up 2 sockets
Aug 10 23:44:03 dist-test-slave-pww0 krb5kdc[5330](info): commencing operation
krb5kdc: starting...
W20260810 23:44:04.504731 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:04 dist-test-slave-pww0 krb5kdc[5330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:04.863191 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:05.229984 5351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:05.230345 5351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:05.230476 5351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:05.238018 5351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:05.238152 5351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:05.238231 5351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:05.238317 5351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:05.238380 5351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:05.248867 5351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:05.250322 5351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:05.251812 5351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:05.264264 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:05.264310 5359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:05.264379 5361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:05.265004 5351 server_base.cc:1034] running on GCE node
I20260810 23:44:05.265905 5351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:05.267520 5351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:05.268802 5351 hybrid_clock.cc:648] HybridClock initialized: now 1786405445268729 us; error 61 us; skew 500 ppm
I20260810 23:44:05.269277 5351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:05 dist-test-slave-pww0 krb5kdc[5330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:05.280802 5351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:05.282541 5351 webserver.cc:460] Webserver started at http://127.0.0.1:42419/ using document root <none> and password file <none>
I20260810 23:44:05.283300 5351 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:05.283439 5351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:05.283807 5351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:05.286985 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "2d916627d7e14cd8b5df82d1563e35d4"
format_stamp: "Formatted at 2026-08-10 23:44:05 on dist-test-slave-pww0"
server_key: "b2e48bd047997a7a9d72a7be65a2fb25"
server_key_iv: "0d06498008c5b682e02dcedbbbef2da8"
server_key_version: "encryptionkey@0"
I20260810 23:44:05.287858 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "2d916627d7e14cd8b5df82d1563e35d4"
format_stamp: "Formatted at 2026-08-10 23:44:05 on dist-test-slave-pww0"
server_key: "b2e48bd047997a7a9d72a7be65a2fb25"
server_key_iv: "0d06498008c5b682e02dcedbbbef2da8"
server_key_version: "encryptionkey@0"
I20260810 23:44:05.293715 5351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:44:05.297705 5377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:05.299443 5351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:44:05.299695 5351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "2d916627d7e14cd8b5df82d1563e35d4"
format_stamp: "Formatted at 2026-08-10 23:44:05 on dist-test-slave-pww0"
server_key: "b2e48bd047997a7a9d72a7be65a2fb25"
server_key_iv: "0d06498008c5b682e02dcedbbbef2da8"
server_key_version: "encryptionkey@0"
I20260810 23:44:05.299906 5351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:05.306066 5351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:05.310803 5351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:05.311173 5351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:05.331171 5351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34339
I20260810 23:44:05.331203 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34339 every 8 connection(s)
I20260810 23:44:05.332899 5351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:05.337141 5466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:05.339732 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5351
I20260810 23:44:05.339919 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:05.340401 30852 external_mini_cluster.cc:1428] Setting key 98cea1fa6db35050b7588d944f88d10f
I20260810 23:44:05.349004 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:05.342411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50501 (local address 127.0.0.1:34339)
0810 23:44:05.343368 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:05.343382 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:05.343430 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:44:05.345236 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:05.345268 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:05.345666 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:05.346052 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:05.347260 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:05.347267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:05.347285 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:05.347312 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:05.347374 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:05.347401 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:05.347666 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:05.347782 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:44:05.347994 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":405,"threads_started":1}
I20260810 23:44:05.349748 5466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4: Bootstrap starting.
I20260810 23:44:05.353447 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:05.353633 5466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:05.355018 5466 log.cc:824] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:05.358824 5466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4: No bootstrap required, opened a new log
I20260810 23:44:05.361568 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:05.358839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45921 (local address 127.0.0.1:34339)
0810 23:44:05.359041 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:05.359047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:05.359563 (+ 516us) server_negotiation.cc:407] Connection header received
0810 23:44:05.359741 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:05.359746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:05.359813 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:05.359926 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:05.360896 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:05.360902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:05.360910 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:05.360931 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:05.360985 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:05.361006 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:05.361135 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:05.361193 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 23:44:05.361264 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:44:05.367393 5466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d916627d7e14cd8b5df82d1563e35d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 23:44:05.367880 5466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:05.368002 5466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d916627d7e14cd8b5df82d1563e35d4, State: Initialized, Role: FOLLOWER
I20260810 23:44:05.368687 5466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d916627d7e14cd8b5df82d1563e35d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 23:44:05.368942 5466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:05.369052 5466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:05.369189 5466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:05.371582 5466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d916627d7e14cd8b5df82d1563e35d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 23:44:05.372128 5466 leader_election.cc:302] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d916627d7e14cd8b5df82d1563e35d4; no voters:
I20260810 23:44:05.372764 5466 leader_election.cc:288] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:05.372947 5482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:05.374161 5482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [term 1 LEADER]: Becoming Leader. State: Replica: 2d916627d7e14cd8b5df82d1563e35d4, State: Running, Role: LEADER
I20260810 23:44:05.374842 5482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d916627d7e14cd8b5df82d1563e35d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 23:44:05.375216 5466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:05.378002 5488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d916627d7e14cd8b5df82d1563e35d4. Latest consensus state: current_term: 1 leader_uuid: "2d916627d7e14cd8b5df82d1563e35d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d916627d7e14cd8b5df82d1563e35d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } } }
I20260810 23:44:05.378505 5488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:05.379016 5483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d916627d7e14cd8b5df82d1563e35d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d916627d7e14cd8b5df82d1563e35d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } } }
I20260810 23:44:05.379335 5483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:05.387269 5497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:05.392257 5497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:05.401980 5497 catalog_manager.cc:1347] Generated new cluster ID: 4b7460b292d741e3ad327c3f9d5c413f
I20260810 23:44:05.402124 5497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:05.439248 5497 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:05.439786 5497 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:05.458362 5497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d916627d7e14cd8b5df82d1563e35d4: Generated new TSK 0
I20260810 23:44:05.459370 5497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:05.471946 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5351
W20260810 23:44:05.473266 5460 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50501
W20260810 23:44:05.474052 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:05.474205 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5351
2026-08-10T23:44:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:05 dist-test-slave-pww0 krb5kdc[5524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:05 dist-test-slave-pww0 krb5kdc[5524](info): set up 2 sockets
Aug 10 23:44:05 dist-test-slave-pww0 krb5kdc[5524](info): commencing operation
krb5kdc: starting...
W20260810 23:44:06.799856 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.004s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:07 dist-test-slave-pww0 krb5kdc[5524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:07.167121 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:07.536826 5544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:07.537185 5544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:07.537338 5544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:07.545063 5544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:07.545228 5544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:07.545313 5544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:07.545414 5544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:07.545473 5544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:07.556501 5544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:07.557965 5544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:07.559439 5544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:07.572693 5555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:07.572652 5559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:07.572626 5561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:07.573935 5544 server_base.cc:1034] running on GCE node
I20260810 23:44:07.574868 5544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:07.576546 5544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:07.577780 5544 hybrid_clock.cc:648] HybridClock initialized: now 1786405447577717 us; error 46 us; skew 500 ppm
I20260810 23:44:07.578169 5544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:07 dist-test-slave-pww0 krb5kdc[5524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:07.581625 5544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:07.583331 5544 webserver.cc:460] Webserver started at http://127.0.0.1:35849/ using document root <none> and password file <none>
I20260810 23:44:07.584110 5544 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:07.584227 5544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:07.584553 5544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:07.587863 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "f00a3e9e115645a293b48e2a3522276e"
format_stamp: "Formatted at 2026-08-10 23:44:07 on dist-test-slave-pww0"
server_key: "ff39e13b49b26a70bd764bed5ec216c2"
server_key_iv: "88d10184cc38ec6df61c054817c353e8"
server_key_version: "encryptionkey@0"
I20260810 23:44:07.588776 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "f00a3e9e115645a293b48e2a3522276e"
format_stamp: "Formatted at 2026-08-10 23:44:07 on dist-test-slave-pww0"
server_key: "ff39e13b49b26a70bd764bed5ec216c2"
server_key_iv: "88d10184cc38ec6df61c054817c353e8"
server_key_version: "encryptionkey@0"
I20260810 23:44:07.594797 5544 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:44:07.598912 5573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:07.600518 5544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:44:07.600733 5544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "f00a3e9e115645a293b48e2a3522276e"
format_stamp: "Formatted at 2026-08-10 23:44:07 on dist-test-slave-pww0"
server_key: "ff39e13b49b26a70bd764bed5ec216c2"
server_key_iv: "88d10184cc38ec6df61c054817c353e8"
server_key_version: "encryptionkey@0"
I20260810 23:44:07.600942 5544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:07.622915 5544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:07.627873 5544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:07.628178 5544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:07.649778 5544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46481
I20260810 23:44:07.649816 5651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46481 every 8 connection(s)
I20260810 23:44:07.651324 5544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:07.654996 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5544
I20260810 23:44:07.655169 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:07.655721 30852 external_mini_cluster.cc:1428] Setting key d513cb116398405a975c61c774e83ce8
I20260810 23:44:07.655818 5655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:07.669384 5655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e: Bootstrap starting.
Aug 10 23:44:07 dist-test-slave-pww0 krb5kdc[5524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:44:07.673029 5655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:07.674268 5655 log.cc:824] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:07.677412 5655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e: No bootstrap required, opened a new log
I20260810 23:44:07.680603 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:07.657845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:44543 (local address 127.0.0.1:46481)
0810 23:44:07.658746 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:07.658761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:07.658810 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:44:07.660964 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:07.660993 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:07.661527 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:07.661988 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:07.663804 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:07.665202 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:07.666593 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:07.667415 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:07.667640 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:07.672307 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:07.672318 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:07.672336 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:07.672384 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:07.675704 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:07.676637 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:07.676644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:07.676652 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:07.676848 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:07.677367 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:07.677371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:07.677373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:07.678079 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:07.678436 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:07.678984 (+ 548us) server_negotiation.cc:299] Negotiation successful
0810 23:44:07.679382 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":386,"threads_started":1}
I20260810 23:44:07.685467 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:07.687104 5655 raft_consensus.cc:348] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00a3e9e115645a293b48e2a3522276e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260810 23:44:07.687640 5655 raft_consensus.cc:374] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:07.687757 5655 raft_consensus.cc:729] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00a3e9e115645a293b48e2a3522276e, State: Initialized, Role: FOLLOWER
I20260810 23:44:07.688493 5655 consensus_queue.cc:260] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00a3e9e115645a293b48e2a3522276e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260810 23:44:07.688860 5655 raft_consensus.cc:388] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:07.689030 5655 raft_consensus.cc:482] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:07.689237 5655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:07.692592 5655 raft_consensus.cc:504] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00a3e9e115645a293b48e2a3522276e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260810 23:44:07.693306 5655 leader_election.cc:302] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f00a3e9e115645a293b48e2a3522276e; no voters:
I20260810 23:44:07.694041 5655 leader_election.cc:288] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:07.694162 5670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:07.695742 5670 raft_consensus.cc:686] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [term 1 LEADER]: Becoming Leader. State: Replica: f00a3e9e115645a293b48e2a3522276e, State: Running, Role: LEADER
I20260810 23:44:07.696548 5655 sys_catalog.cc:564] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:07.696455 5670 consensus_queue.cc:237] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00a3e9e115645a293b48e2a3522276e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260810 23:44:07.699783 5671 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f00a3e9e115645a293b48e2a3522276e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00a3e9e115645a293b48e2a3522276e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260810 23:44:07.700196 5671 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:07.701990 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f00a3e9e115645a293b48e2a3522276e. Latest consensus state: current_term: 1 leader_uuid: "f00a3e9e115645a293b48e2a3522276e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00a3e9e115645a293b48e2a3522276e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260810 23:44:07.702382 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:07.705996 5685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:07.711479 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:07.691042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:49325 (local address 127.0.0.1:46481)
0810 23:44:07.691293 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:07.691301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:07.691771 (+ 470us) server_negotiation.cc:407] Connection header received
0810 23:44:07.691933 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:07.691937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:07.692007 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:07.692092 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:07.694084 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:07.695409 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:07.697362 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:07.698091 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:07.699069 (+ 978us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:07.702122 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:07.702131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:07.702136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:07.702195 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:07.706660 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:07.707942 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:07.707949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:07.707952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:07.708137 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:07.708793 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:07.708798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:07.708801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:07.709276 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:07.709594 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:07.710892 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0810 23:44:07.711094 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:44:07.712610 5685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:07.724594 5685 catalog_manager.cc:1347] Generated new cluster ID: 58d6f7d6f399457b969ead428172a5b6
I20260810 23:44:07.724732 5685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:07.745906 5685 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:07.747370 5685 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:07.756835 5685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f00a3e9e115645a293b48e2a3522276e: Generated new TSK 0
I20260810 23:44:07.757889 5685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:07.772018 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5544
I20260810 23:44:07.773630 5644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:07.868551 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5544
W20260810 23:44:07.894402 5521 connection.cc:535] client connection to 127.0.0.1:46481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46481): Connection reset by peer (error 104)
2026-08-10T23:44:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:07 dist-test-slave-pww0 krb5kdc[5709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:07 dist-test-slave-pww0 krb5kdc[5709](info): set up 2 sockets
Aug 10 23:44:07 dist-test-slave-pww0 krb5kdc[5709](info): commencing operation
krb5kdc: starting...
W20260810 23:44:09.193372 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:09 dist-test-slave-pww0 krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:09.556200 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:09.925597 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:09.925937 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:09.926081 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:09.933650 5729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:09.933796 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:09.933851 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:09.933938 5729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:09.933997 5729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:09.944429 5729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:09.945852 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:09.947294 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:09.959892 5738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:09.959933 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:09.959935 5746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:09.960974 5729 server_base.cc:1034] running on GCE node
I20260810 23:44:09.961877 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:09.963502 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:09.964756 5729 hybrid_clock.cc:648] HybridClock initialized: now 1786405449964703 us; error 50 us; skew 500 ppm
I20260810 23:44:09.965157 5729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:09 dist-test-slave-pww0 krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:09.968627 5729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:09.970358 5729 webserver.cc:460] Webserver started at http://127.0.0.1:35003/ using document root <none> and password file <none>
I20260810 23:44:09.971153 5729 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:09.971290 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:09.971663 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:09.974913 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4"
format_stamp: "Formatted at 2026-08-10 23:44:09 on dist-test-slave-pww0"
server_key: "c68a71ce20e906e72eee73f85f238a4e"
server_key_iv: "4b5f368dab442fb851d14b4d876f8b66"
server_key_version: "encryptionkey@0"
I20260810 23:44:09.975816 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4"
format_stamp: "Formatted at 2026-08-10 23:44:09 on dist-test-slave-pww0"
server_key: "c68a71ce20e906e72eee73f85f238a4e"
server_key_iv: "4b5f368dab442fb851d14b4d876f8b66"
server_key_version: "encryptionkey@0"
I20260810 23:44:09.981588 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:44:09.985678 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:09.987260 5729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:44:09.987486 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4"
format_stamp: "Formatted at 2026-08-10 23:44:09 on dist-test-slave-pww0"
server_key: "c68a71ce20e906e72eee73f85f238a4e"
server_key_iv: "4b5f368dab442fb851d14b4d876f8b66"
server_key_version: "encryptionkey@0"
I20260810 23:44:09.987709 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:10.017043 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:10.021669 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:10.021981 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:10.041581 5729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46755
I20260810 23:44:10.041646 5861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46755 every 8 connection(s)
I20260810 23:44:10.043092 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:10.043859 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5729
I20260810 23:44:10.044049 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:10.044728 30852 external_mini_cluster.cc:1428] Setting key eca05be40ac32ccd04c459d27509a064
I20260810 23:44:10.047909 5865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:10.058876 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:10.046813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:49921 (local address 127.0.0.1:46755)
0810 23:44:10.047700 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:10.047713 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:10.047760 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:44:10.049549 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:10.049575 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:10.050007 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:10.050438 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:10.051079 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:10.052519 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:10.054000 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:10.054755 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:10.054945 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:10.056337 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:10.056346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:10.056362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:10.056995 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:10.057084 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:10.057125 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:10.057289 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:10.057622 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 23:44:10.057811 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":345,"threads_started":1}
I20260810 23:44:10.059700 5865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4: Bootstrap starting.
I20260810 23:44:10.063704 5865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:10.063700 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:10.064818 5865 log.cc:824] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:10.068279 5865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4: No bootstrap required, opened a new log
I20260810 23:44:10.075114 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:10.068474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:43911 (local address 127.0.0.1:46755)
0810 23:44:10.068637 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:10.068643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:10.069160 (+ 517us) server_negotiation.cc:407] Connection header received
0810 23:44:10.069363 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:10.069367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:10.069434 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:10.069522 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:10.070099 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:10.070935 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:10.072420 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:10.072963 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:10.073093 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:10.074310 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:10.074315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:10.074318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:10.074339 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:10.074402 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:10.074422 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:10.074537 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:10.074745 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 23:44:10.074806 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:44:10.077075 5865 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46755 } }
I20260810 23:44:10.077512 5865 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:10.077617 5865 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0aea5a32e3402ebb1ea1ceb00d62f4, State: Initialized, Role: FOLLOWER
I20260810 23:44:10.078342 5865 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46755 } }
I20260810 23:44:10.078681 5865 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:10.078835 5865 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:10.079026 5865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:10.081506 5865 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46755 } }
I20260810 23:44:10.082110 5865 leader_election.cc:302] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0aea5a32e3402ebb1ea1ceb00d62f4; no voters:
I20260810 23:44:10.082770 5865 leader_election.cc:288] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:10.082919 5885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:10.083752 5885 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [term 1 LEADER]: Becoming Leader. State: Replica: fc0aea5a32e3402ebb1ea1ceb00d62f4, State: Running, Role: LEADER
I20260810 23:44:10.084390 5885 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46755 } }
I20260810 23:44:10.085088 5865 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:10.087445 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0aea5a32e3402ebb1ea1ceb00d62f4. Latest consensus state: current_term: 1 leader_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46755 } } }
I20260810 23:44:10.087957 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:10.088171 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0aea5a32e3402ebb1ea1ceb00d62f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46755 } } }
I20260810 23:44:10.088558 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:10.090579 5898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:10.095914 5898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:10.105754 5898 catalog_manager.cc:1347] Generated new cluster ID: e434b7f9b81643bb8f84035c454bc751
I20260810 23:44:10.105896 5898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:10.117676 5898 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:10.119661 5898 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:10.131690 5898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc0aea5a32e3402ebb1ea1ceb00d62f4: Generated new TSK 0
I20260810 23:44:10.132916 5898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:10.150796 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5729
W20260810 23:44:10.152596 5856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:49921
W20260810 23:44:10.153466 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5729: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:10.153626 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5729
W20260810 23:44:10.176677 5706 connection.cc:535] client connection to 127.0.0.1:46755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46755): Connection reset by peer (error 104)
2026-08-10T23:44:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:10 dist-test-slave-pww0 krb5kdc[5932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:10 dist-test-slave-pww0 krb5kdc[5932](info): set up 2 sockets
Aug 10 23:44:10 dist-test-slave-pww0 krb5kdc[5932](info): commencing operation
krb5kdc: starting...
W20260810 23:44:11.471096 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:11 dist-test-slave-pww0 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:11.824657 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:12.182404 5957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:12.182734 5957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:12.182876 5957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:12.190541 5957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:12.190681 5957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:12.190748 5957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:12.190837 5957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:12.190896 5957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:12.201521 5957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:12.202937 5957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:12.204345 5957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:12.216542 5964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:12.216507 5968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:12.216468 5965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:12.217367 5957 server_base.cc:1034] running on GCE node
I20260810 23:44:12.218246 5957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:12.219869 5957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:12.221082 5957 hybrid_clock.cc:648] HybridClock initialized: now 1786405452221034 us; error 36 us; skew 500 ppm
I20260810 23:44:12.221447 5957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:12 dist-test-slave-pww0 krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:12.227890 5957 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:12.229656 5957 webserver.cc:460] Webserver started at http://127.0.0.1:44107/ using document root <none> and password file <none>
I20260810 23:44:12.230400 5957 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:12.230531 5957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:12.230868 5957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:12.234064 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "f3fa236fcfb34021b60261f991c8eec6"
format_stamp: "Formatted at 2026-08-10 23:44:12 on dist-test-slave-pww0"
server_key: "318bafb7cd33fc1af65276a4bc1b420a"
server_key_iv: "14ccfc7bf01413922e8fd8007c37d962"
server_key_version: "encryptionkey@0"
I20260810 23:44:12.234885 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "f3fa236fcfb34021b60261f991c8eec6"
format_stamp: "Formatted at 2026-08-10 23:44:12 on dist-test-slave-pww0"
server_key: "318bafb7cd33fc1af65276a4bc1b420a"
server_key_iv: "14ccfc7bf01413922e8fd8007c37d962"
server_key_version: "encryptionkey@0"
I20260810 23:44:12.240531 5957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:44:12.244316 5980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:12.245782 5957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:44:12.245965 5957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "f3fa236fcfb34021b60261f991c8eec6"
format_stamp: "Formatted at 2026-08-10 23:44:12 on dist-test-slave-pww0"
server_key: "318bafb7cd33fc1af65276a4bc1b420a"
server_key_iv: "14ccfc7bf01413922e8fd8007c37d962"
server_key_version: "encryptionkey@0"
I20260810 23:44:12.246174 5957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:12.252640 5957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:12.256800 5957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:12.257115 5957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:12.276016 5957 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36607
I20260810 23:44:12.276067 6067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36607 every 8 connection(s)
I20260810 23:44:12.277493 5957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:12.279956 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 5957
I20260810 23:44:12.280108 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:12.280542 30852 external_mini_cluster.cc:1428] Setting key 1ba1859de719d630dc785c8e96316820
I20260810 23:44:12.281791 6068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:12.288796 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:12.282512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:35159 (local address 127.0.0.1:36607)
0810 23:44:12.283326 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:12.283339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:12.283384 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:44:12.285029 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:12.285053 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:12.285442 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:12.285823 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:12.286992 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:12.286999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:12.287012 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:12.287027 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:12.287108 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:12.287130 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:12.287319 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:12.287422 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 23:44:12.287671 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":340,"threads_started":1}
I20260810 23:44:12.292749 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:12.295817 6068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6: Bootstrap starting.
I20260810 23:44:12.300029 6068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:12.300038 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:12.297482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:58581 (local address 127.0.0.1:36607)
0810 23:44:12.297715 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:12.297722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:12.297965 (+ 243us) server_negotiation.cc:407] Connection header received
0810 23:44:12.298145 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:12.298152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:12.298223 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:12.298356 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:12.299232 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:12.299238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:12.299243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:12.299264 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:12.299317 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:12.299341 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:12.299498 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:12.299603 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 23:44:12.299671 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 23:44:12.301492 6068 log.cc:824] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:12.304574 6068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6: No bootstrap required, opened a new log
I20260810 23:44:12.312233 6068 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fa236fcfb34021b60261f991c8eec6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260810 23:44:12.312652 6068 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:12.312768 6068 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fa236fcfb34021b60261f991c8eec6, State: Initialized, Role: FOLLOWER
I20260810 23:44:12.313483 6068 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fa236fcfb34021b60261f991c8eec6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260810 23:44:12.313778 6068 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:12.313930 6068 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:12.314091 6068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:12.316440 6068 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fa236fcfb34021b60261f991c8eec6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260810 23:44:12.316996 6068 leader_election.cc:302] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3fa236fcfb34021b60261f991c8eec6; no voters:
I20260810 23:44:12.317592 6068 leader_election.cc:288] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:12.317731 6089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:12.318423 6089 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [term 1 LEADER]: Becoming Leader. State: Replica: f3fa236fcfb34021b60261f991c8eec6, State: Running, Role: LEADER
I20260810 23:44:12.318985 6089 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fa236fcfb34021b60261f991c8eec6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260810 23:44:12.319747 6068 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:12.321852 6091 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3fa236fcfb34021b60261f991c8eec6. Latest consensus state: current_term: 1 leader_uuid: "f3fa236fcfb34021b60261f991c8eec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fa236fcfb34021b60261f991c8eec6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } } }
I20260810 23:44:12.321797 6090 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3fa236fcfb34021b60261f991c8eec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fa236fcfb34021b60261f991c8eec6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } } }
I20260810 23:44:12.322371 6090 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:12.323186 6091 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:12.324856 6097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:12.330188 6097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:12.340261 6097 catalog_manager.cc:1347] Generated new cluster ID: 2f596ba8354941ea9bf4f66f750da5f2
I20260810 23:44:12.340387 6097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:12.361497 6097 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:12.362000 6097 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:12.368114 6097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3fa236fcfb34021b60261f991c8eec6: Generated new TSK 0
I20260810 23:44:12.369120 6097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:12.386853 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5957
W20260810 23:44:12.388438 6059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:35159
W20260810 23:44:12.389226 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 5957: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:12.389401 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 5957
2026-08-10T23:44:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:12 dist-test-slave-pww0 krb5kdc[6127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:12 dist-test-slave-pww0 krb5kdc[6127](info): set up 2 sockets
Aug 10 23:44:12 dist-test-slave-pww0 krb5kdc[6127](info): commencing operation
krb5kdc: starting...
W20260810 23:44:13.718494 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.004s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:13 dist-test-slave-pww0 krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:14.078241 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:14.453251 6155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:14.453599 6155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:14.453744 6155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:14.461488 6155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:14.461658 6155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:14.461745 6155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:14.461840 6155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:14.461905 6155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:14.472975 6155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:14.474507 6155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:14.476037 6155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:14.489217 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:14.489329 6175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:14.489289 6165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:14.490506 6155 server_base.cc:1034] running on GCE node
I20260810 23:44:14.491492 6155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:14.493242 6155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:14.494488 6155 hybrid_clock.cc:648] HybridClock initialized: now 1786405454494421 us; error 56 us; skew 500 ppm
I20260810 23:44:14.494887 6155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:14 dist-test-slave-pww0 krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:14.498404 6155 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:14.500222 6155 webserver.cc:460] Webserver started at http://127.0.0.1:35303/ using document root <none> and password file <none>
I20260810 23:44:14.501017 6155 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:14.501160 6155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:14.501499 6155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:14.504869 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "dd087169ce054755a7ce9e7fd472a36a"
format_stamp: "Formatted at 2026-08-10 23:44:14 on dist-test-slave-pww0"
server_key: "d1358a848921a2021605211b5113213f"
server_key_iv: "b0469bb192ea45cda77a8f88ae3ce4f3"
server_key_version: "encryptionkey@0"
I20260810 23:44:14.505726 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "dd087169ce054755a7ce9e7fd472a36a"
format_stamp: "Formatted at 2026-08-10 23:44:14 on dist-test-slave-pww0"
server_key: "d1358a848921a2021605211b5113213f"
server_key_iv: "b0469bb192ea45cda77a8f88ae3ce4f3"
server_key_version: "encryptionkey@0"
I20260810 23:44:14.511641 6155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:44:14.515568 6184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:14.517074 6155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:44:14.517287 6155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "dd087169ce054755a7ce9e7fd472a36a"
format_stamp: "Formatted at 2026-08-10 23:44:14 on dist-test-slave-pww0"
server_key: "d1358a848921a2021605211b5113213f"
server_key_iv: "b0469bb192ea45cda77a8f88ae3ce4f3"
server_key_version: "encryptionkey@0"
I20260810 23:44:14.517496 6155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:14.544606 6155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:14.549611 6155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:14.549934 6155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:14.569018 6155 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45849
I20260810 23:44:14.569051 6271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45849 every 8 connection(s)
I20260810 23:44:14.570539 6155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:14.574510 6274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:14.575304 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6155
I20260810 23:44:14.575551 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:14.576133 30852 external_mini_cluster.cc:1428] Setting key fb1fa0aea30b88283c2f0b317b390b15
I20260810 23:44:14.589275 6274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a: Bootstrap starting.
Aug 10 23:44:14 dist-test-slave-pww0 krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:44:14.593410 6274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:14.594748 6274 log.cc:824] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:14.598286 6274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a: No bootstrap required, opened a new log
I20260810 23:44:14.602239 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:14.578111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:58777 (local address 127.0.0.1:45849)
0810 23:44:14.578932 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:14.578946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:14.578991 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:44:14.581013 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:14.581040 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:14.581549 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:14.581922 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:14.583932 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:14.585378 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:14.587276 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:14.588113 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:14.588252 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:14.593513 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:14.593523 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:14.593553 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:14.593617 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:14.597155 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:14.598113 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:14.598121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:14.598128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:14.598322 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:14.598868 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:14.598872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:14.598874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:14.599489 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:14.599867 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:14.600509 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 23:44:14.601024 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":383,"threads_started":1}
I20260810 23:44:14.607403 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:14.608583 6274 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd087169ce054755a7ce9e7fd472a36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 23:44:14.609057 6274 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:14.609176 6274 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd087169ce054755a7ce9e7fd472a36a, State: Initialized, Role: FOLLOWER
I20260810 23:44:14.609894 6274 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd087169ce054755a7ce9e7fd472a36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 23:44:14.610301 6274 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:14.610507 6274 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:14.610733 6274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:14.613947 6274 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd087169ce054755a7ce9e7fd472a36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 23:44:14.614583 6274 leader_election.cc:302] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd087169ce054755a7ce9e7fd472a36a; no voters:
I20260810 23:44:14.615347 6274 leader_election.cc:288] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:14.615492 6296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:14.618168 6274 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:14.618712 6296 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [term 1 LEADER]: Becoming Leader. State: Replica: dd087169ce054755a7ce9e7fd472a36a, State: Running, Role: LEADER
I20260810 23:44:14.619834 6296 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd087169ce054755a7ce9e7fd472a36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 23:44:14.624289 6297 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd087169ce054755a7ce9e7fd472a36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd087169ce054755a7ce9e7fd472a36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260810 23:44:14.624861 6297 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:14.625430 6301 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd087169ce054755a7ce9e7fd472a36a. Latest consensus state: current_term: 1 leader_uuid: "dd087169ce054755a7ce9e7fd472a36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd087169ce054755a7ce9e7fd472a36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260810 23:44:14.625814 6301 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:14.625957 6315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:14.631322 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:14.612319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:60009 (local address 127.0.0.1:45849)
0810 23:44:14.612499 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:14.612506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:14.612965 (+ 459us) server_negotiation.cc:407] Connection header received
0810 23:44:14.613158 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:14.613164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:14.613248 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:14.613337 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:14.614896 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:14.616057 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:14.617598 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:14.618131 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:14.618267 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:14.621728 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:14.621736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:14.621740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:14.621784 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:14.626034 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:14.627364 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:14.627370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:14.627375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:14.627571 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:14.628229 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:14.628234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:14.628251 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:14.628766 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:14.630490 (+ 1724us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:14.630705 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:44:14.630929 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:44:14.637315 6315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:14.649317 6315 catalog_manager.cc:1347] Generated new cluster ID: d23ecbf0a7514d7190b0f39ebf3bca7e
I20260810 23:44:14.649526 6315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:14.657958 6315 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:14.660009 6315 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:14.669361 6315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd087169ce054755a7ce9e7fd472a36a: Generated new TSK 0
I20260810 23:44:14.670276 6315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:14.679919 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6155
I20260810 23:44:14.681744 6266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:14.774487 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6155
W20260810 23:44:14.797278 6125 connection.cc:535] client connection to 127.0.0.1:45849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45849): Connection reset by peer (error 104)
2026-08-10T23:44:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:14 dist-test-slave-pww0 krb5kdc[6339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:14 dist-test-slave-pww0 krb5kdc[6339](info): set up 2 sockets
Aug 10 23:44:14 dist-test-slave-pww0 krb5kdc[6339](info): commencing operation
krb5kdc: starting...
W20260810 23:44:16.091611 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:16 dist-test-slave-pww0 krb5kdc[6339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:16.448160 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:16.804630 6361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:16.804960 6361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:16.805064 6361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:16.812525 6361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:16.812660 6361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:16.812716 6361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:16.812777 6361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:16.812822 6361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:16.823698 6361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:16.825065 6361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:16.826385 6361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:16.838469 6369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:16.838558 6371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:16.838539 6368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:16.839753 6361 server_base.cc:1034] running on GCE node
I20260810 23:44:16.840698 6361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:16.842334 6361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:16.843560 6361 hybrid_clock.cc:648] HybridClock initialized: now 1786405456843501 us; error 50 us; skew 500 ppm
I20260810 23:44:16.843936 6361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:16 dist-test-slave-pww0 krb5kdc[6339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:16.847261 6361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:16.849004 6361 webserver.cc:460] Webserver started at http://127.0.0.1:41847/ using document root <none> and password file <none>
I20260810 23:44:16.849743 6361 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:16.849872 6361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:16.850178 6361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:16.853372 6361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "dfb56cf0c08d4f459ab4be0b8129e40d"
format_stamp: "Formatted at 2026-08-10 23:44:16 on dist-test-slave-pww0"
server_key: "cc2ac623c0bbbbc168b77fb87e78e68b"
server_key_iv: "41de1525c178b9e14a4140bc9c0246c1"
server_key_version: "encryptionkey@0"
I20260810 23:44:16.854282 6361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "dfb56cf0c08d4f459ab4be0b8129e40d"
format_stamp: "Formatted at 2026-08-10 23:44:16 on dist-test-slave-pww0"
server_key: "cc2ac623c0bbbbc168b77fb87e78e68b"
server_key_iv: "41de1525c178b9e14a4140bc9c0246c1"
server_key_version: "encryptionkey@0"
I20260810 23:44:16.860085 6361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260810 23:44:16.863989 6385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:16.865522 6361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:44:16.865720 6361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "dfb56cf0c08d4f459ab4be0b8129e40d"
format_stamp: "Formatted at 2026-08-10 23:44:16 on dist-test-slave-pww0"
server_key: "cc2ac623c0bbbbc168b77fb87e78e68b"
server_key_iv: "41de1525c178b9e14a4140bc9c0246c1"
server_key_version: "encryptionkey@0"
I20260810 23:44:16.865927 6361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:16.885156 6361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:16.889715 6361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:16.890013 6361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:16.909194 6361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36455
I20260810 23:44:16.909222 6465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36455 every 8 connection(s)
I20260810 23:44:16.910737 6361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:16.913843 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6361
I20260810 23:44:16.914053 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:16.914618 30852 external_mini_cluster.cc:1428] Setting key e600ec09ea9191eb429d55925452cca1
I20260810 23:44:16.915236 6466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:16.931648 6466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d: Bootstrap starting.
I20260810 23:44:16.931972 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:16.916826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:36483 (local address 127.0.0.1:36455)
0810 23:44:16.917825 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:16.917841 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:16.918210 (+ 369us) server_negotiation.cc:407] Connection header received
0810 23:44:16.920307 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:16.920346 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:16.921020 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:16.921554 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:16.922193 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:16.924470 (+ 2277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:16.926028 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:16.927143 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:16.927365 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:16.928812 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:16.928827 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:16.928850 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:16.929636 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:16.929752 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:16.929794 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:16.929962 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:16.930304 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:44:16.930512 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":410,"threads_started":1}
I20260810 23:44:16.935227 6466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:16.936218 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:16.936636 6466 log.cc:824] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:16.940236 6466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d: No bootstrap required, opened a new log
I20260810 23:44:16.948468 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:16.941385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:48389 (local address 127.0.0.1:36455)
0810 23:44:16.941584 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:16.941590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:16.942126 (+ 536us) server_negotiation.cc:407] Connection header received
0810 23:44:16.942299 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:16.942303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:16.942379 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:16.942468 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:16.943053 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:16.943903 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:16.945818 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:16.946295 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:16.946393 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:16.947660 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:16.947665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:16.947669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:16.947687 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:16.947748 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:16.947764 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:16.947852 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:16.948060 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 23:44:16.948112 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:44:16.950390 6466 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260810 23:44:16.950815 6466 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:16.950918 6466 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb56cf0c08d4f459ab4be0b8129e40d, State: Initialized, Role: FOLLOWER
I20260810 23:44:16.951571 6466 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260810 23:44:16.951813 6466 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:16.951947 6466 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:16.952105 6466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:16.954468 6466 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260810 23:44:16.955090 6466 leader_election.cc:302] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfb56cf0c08d4f459ab4be0b8129e40d; no voters:
I20260810 23:44:16.955763 6466 leader_election.cc:288] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:16.955910 6482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:16.956615 6482 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [term 1 LEADER]: Becoming Leader. State: Replica: dfb56cf0c08d4f459ab4be0b8129e40d, State: Running, Role: LEADER
I20260810 23:44:16.957181 6482 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } }
I20260810 23:44:16.957902 6466 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:16.959919 6486 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb56cf0c08d4f459ab4be0b8129e40d. Latest consensus state: current_term: 1 leader_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } } }
I20260810 23:44:16.960273 6486 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:16.960747 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb56cf0c08d4f459ab4be0b8129e40d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36455 } } }
I20260810 23:44:16.961077 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:16.967993 6494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:16.973088 6494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:16.982674 6494 catalog_manager.cc:1347] Generated new cluster ID: 10a251cf435e44a2a0191b9c1c074ec3
I20260810 23:44:16.982823 6494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:17.003118 6494 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:17.004832 6494 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:17.020949 6494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfb56cf0c08d4f459ab4be0b8129e40d: Generated new TSK 0
I20260810 23:44:17.022210 6494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:17.032939 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6361
W20260810 23:44:17.034345 6458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:36483
W20260810 23:44:17.035264 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6361: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:17.035444 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6361
W20260810 23:44:17.058959 6333 connection.cc:535] client connection to 127.0.0.1:36455 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36455): Connection reset by peer (error 104)
2026-08-10T23:44:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:17 dist-test-slave-pww0 krb5kdc[6513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:17 dist-test-slave-pww0 krb5kdc[6513](info): set up 2 sockets
Aug 10 23:44:17 dist-test-slave-pww0 krb5kdc[6513](info): commencing operation
krb5kdc: starting...
W20260810 23:44:18.362960 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.002s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:18 dist-test-slave-pww0 krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:44:18.715294 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:19.070065 6541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:19.070410 6541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:19.070509 6541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:19.078135 6541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:19.078276 6541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:19.078325 6541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:19.078387 6541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:19.078433 6541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:19.088755 6541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:19.090116 6541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:19.091500 6541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:19.104118 6550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:19.104077 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:19.104071 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:19.105060 6541 server_base.cc:1034] running on GCE node
I20260810 23:44:19.105932 6541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:19.107578 6541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:19.108837 6541 hybrid_clock.cc:648] HybridClock initialized: now 1786405459108764 us; error 60 us; skew 500 ppm
I20260810 23:44:19.109241 6541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:19 dist-test-slave-pww0 krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:19.116412 6541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:44:19.118172 6541 webserver.cc:460] Webserver started at http://127.0.0.1:36715/ using document root <none> and password file <none>
I20260810 23:44:19.118988 6541 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:19.119155 6541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:19.119536 6541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:19.122751 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "9671680814ab460cafcfabc9396e93b0"
format_stamp: "Formatted at 2026-08-10 23:44:19 on dist-test-slave-pww0"
server_key: "3501fab155a8b2e5fa1a6c6109fa096b"
server_key_iv: "96a840b652b4b3046d583a1522e107dc"
server_key_version: "encryptionkey@0"
I20260810 23:44:19.123657 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "9671680814ab460cafcfabc9396e93b0"
format_stamp: "Formatted at 2026-08-10 23:44:19 on dist-test-slave-pww0"
server_key: "3501fab155a8b2e5fa1a6c6109fa096b"
server_key_iv: "96a840b652b4b3046d583a1522e107dc"
server_key_version: "encryptionkey@0"
I20260810 23:44:19.130502 6541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:44:19.138545 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:19.140905 6541 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260810 23:44:19.141163 6541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "9671680814ab460cafcfabc9396e93b0"
format_stamp: "Formatted at 2026-08-10 23:44:19 on dist-test-slave-pww0"
server_key: "3501fab155a8b2e5fa1a6c6109fa096b"
server_key_iv: "96a840b652b4b3046d583a1522e107dc"
server_key_version: "encryptionkey@0"
I20260810 23:44:19.141408 6541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:19.147912 6541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:19.152737 6541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:19.153086 6541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:19.173338 6541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43843
I20260810 23:44:19.173380 6649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43843 every 8 connection(s)
I20260810 23:44:19.175022 6541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:19.179272 6650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:19.180264 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6541
I20260810 23:44:19.180451 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:19.180902 30852 external_mini_cluster.cc:1428] Setting key 1f2bd09b7f8298cfd030464b23d02341
I20260810 23:44:19.190636 6657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:19.182806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:41607 (local address 127.0.0.1:43843)
0810 23:44:19.183704 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:19.183717 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:19.183764 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:44:19.185667 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:19.185703 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:19.186192 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:19.186716 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:19.188145 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:19.188155 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:19.188175 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:19.188198 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:19.188286 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:19.188323 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:19.188567 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:19.189056 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 23:44:19.189275 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":364,"threads_started":1}
I20260810 23:44:19.194787 6650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0: Bootstrap starting.
I20260810 23:44:19.195262 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:19.198290 6650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:19.199553 6650 log.cc:824] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:19.202654 6657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:19.199795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:47687 (local address 127.0.0.1:43843)
0810 23:44:19.199959 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:19.199963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:19.200577 (+ 614us) server_negotiation.cc:407] Connection header received
0810 23:44:19.200800 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:19.200804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:19.200864 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:19.200953 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:19.201950 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:19.201956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:19.201960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:19.201980 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:19.202031 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:19.202053 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:19.202188 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:19.202302 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 23:44:19.202370 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:44:19.203318 6650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0: No bootstrap required, opened a new log
I20260810 23:44:19.211756 6650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9671680814ab460cafcfabc9396e93b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260810 23:44:19.212320 6650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:19.212421 6650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9671680814ab460cafcfabc9396e93b0, State: Initialized, Role: FOLLOWER
I20260810 23:44:19.213039 6650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9671680814ab460cafcfabc9396e93b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260810 23:44:19.213289 6650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:19.213399 6650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:19.213524 6650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:19.215989 6650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9671680814ab460cafcfabc9396e93b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260810 23:44:19.216635 6650 leader_election.cc:302] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9671680814ab460cafcfabc9396e93b0; no voters:
I20260810 23:44:19.217252 6650 leader_election.cc:288] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:19.217401 6673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:19.218075 6673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [term 1 LEADER]: Becoming Leader. State: Replica: 9671680814ab460cafcfabc9396e93b0, State: Running, Role: LEADER
I20260810 23:44:19.218677 6673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9671680814ab460cafcfabc9396e93b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } }
I20260810 23:44:19.219362 6650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:19.221421 6676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9671680814ab460cafcfabc9396e93b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9671680814ab460cafcfabc9396e93b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } } }
I20260810 23:44:19.221709 6677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9671680814ab460cafcfabc9396e93b0. Latest consensus state: current_term: 1 leader_uuid: "9671680814ab460cafcfabc9396e93b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9671680814ab460cafcfabc9396e93b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43843 } } }
I20260810 23:44:19.221897 6676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:19.221963 6677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0 [sys.catalog]: This master's current role is: LEADER
W20260810 23:44:19.232401 6708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:44:19.232547 6708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:44:19.232714 6691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:19.234431 6691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:19.246035 6691 catalog_manager.cc:1347] Generated new cluster ID: ca65dc8ec69f4254b86f4707e8d89a4d
I20260810 23:44:19.246292 6691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:19.263760 6691 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:19.264185 6691 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:19.276582 6691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9671680814ab460cafcfabc9396e93b0: Generated new TSK 0
I20260810 23:44:19.277594 6691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:19.289331 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6541
W20260810 23:44:19.290737 6643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:41607
W20260810 23:44:19.291481 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:19.291635 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6541
2026-08-10T23:44:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2262 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:19 dist-test-slave-pww0 krb5kdc[6724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:19 dist-test-slave-pww0 krb5kdc[6724](info): set up 2 sockets
Aug 10 23:44:19 dist-test-slave-pww0 krb5kdc[6724](info): commencing operation
krb5kdc: starting...
W20260810 23:44:20.618724 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:20 dist-test-slave-pww0 krb5kdc[6724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:20.979669 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37887
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:21.344003 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:21.344383 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:21.344491 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:21.351996 6744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:21.352133 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:21.352190 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:21.352254 6744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:21.352298 6744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:21.362588 6744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:37887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:21.363978 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:21.365389 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:21.377517 6758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:21.377629 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:21.377558 6761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:21.378340 6744 server_base.cc:1034] running on GCE node
I20260810 23:44:21.379192 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:21.380856 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:21.382113 6744 hybrid_clock.cc:648] HybridClock initialized: now 1786405461382063 us; error 48 us; skew 500 ppm
I20260810 23:44:21.382531 6744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:21 dist-test-slave-pww0 krb5kdc[6724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405461, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:21.385936 6744 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:21.387619 6744 webserver.cc:460] Webserver started at http://127.30.33.62:38053/ using document root <none> and password file <none>
I20260810 23:44:21.388388 6744 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:21.388517 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:21.388820 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:21.392089 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "1df7a89860364bd3b23f0d8f94d727b8"
format_stamp: "Formatted at 2026-08-10 23:44:21 on dist-test-slave-pww0"
server_key: "4156aa9a1fdb8295eb8bfd5618b33bb0"
server_key_iv: "d24f69ce831967f06c2b189d4cd7a517"
server_key_version: "encryptionkey@0"
I20260810 23:44:21.392920 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "1df7a89860364bd3b23f0d8f94d727b8"
format_stamp: "Formatted at 2026-08-10 23:44:21 on dist-test-slave-pww0"
server_key: "4156aa9a1fdb8295eb8bfd5618b33bb0"
server_key_iv: "d24f69ce831967f06c2b189d4cd7a517"
server_key_version: "encryptionkey@0"
I20260810 23:44:21.398666 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:44:21.402453 6774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:21.403983 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:44:21.404183 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "1df7a89860364bd3b23f0d8f94d727b8"
format_stamp: "Formatted at 2026-08-10 23:44:21 on dist-test-slave-pww0"
server_key: "4156aa9a1fdb8295eb8bfd5618b33bb0"
server_key_iv: "d24f69ce831967f06c2b189d4cd7a517"
server_key_version: "encryptionkey@0"
I20260810 23:44:21.404402 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:21.422245 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:21.426846 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:21.427139 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:21.445703 6744 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:37887
I20260810 23:44:21.445739 6885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:37887 every 8 connection(s)
I20260810 23:44:21.447176 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:21.451148 6887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:21.456664 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6744
I20260810 23:44:21.456871 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:21.457427 30852 external_mini_cluster.cc:1428] Setting key 6b7c80b035f1a8bfc1a1d77c3299119a
I20260810 23:44:21.463404 6887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8: Bootstrap starting.
I20260810 23:44:21.466951 6887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:21.468132 6887 log.cc:824] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:21.471951 6887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8: No bootstrap required, opened a new log
Aug 10 23:44:21 dist-test-slave-pww0 krb5kdc[6724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:21.481132 6887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df7a89860364bd3b23f0d8f94d727b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37887 } }
I20260810 23:44:21.481540 6887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:21.481647 6887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df7a89860364bd3b23f0d8f94d727b8, State: Initialized, Role: FOLLOWER
I20260810 23:44:21.482098 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:21.459601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54035 (local address 127.30.33.62:37887)
0810 23:44:21.460519 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:21.460533 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:21.460581 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:44:21.462449 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:21.462475 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:21.462931 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:21.463333 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:21.464994 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:21.466396 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:21.467860 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:21.468653 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:21.468897 (+ 244us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:21.474097 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:21.474107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:21.474124 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:21.474166 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:21.477368 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:21.478461 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:21.478468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:21.478475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:21.478651 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:21.479142 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:21.479146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:21.479148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:21.479811 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:21.480057 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:21.480546 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 23:44:21.480894 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":399,"threads_started":1}
I20260810 23:44:21.482268 6887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df7a89860364bd3b23f0d8f94d727b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37887 } }
I20260810 23:44:21.482555 6887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:21.482702 6887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:21.482882 6887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:21.485575 6887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df7a89860364bd3b23f0d8f94d727b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37887 } }
I20260810 23:44:21.486160 6887 leader_election.cc:302] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df7a89860364bd3b23f0d8f94d727b8; no voters:
I20260810 23:44:21.486500 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:21.486835 6887 leader_election.cc:288] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:21.486934 6900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:21.489086 6887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:21.496084 6900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1df7a89860364bd3b23f0d8f94d727b8, State: Running, Role: LEADER
I20260810 23:44:21.497006 6900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df7a89860364bd3b23f0d8f94d727b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37887 } }
I20260810 23:44:21.500065 6904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df7a89860364bd3b23f0d8f94d727b8. Latest consensus state: current_term: 1 leader_uuid: "1df7a89860364bd3b23f0d8f94d727b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df7a89860364bd3b23f0d8f94d727b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37887 } } }
I20260810 23:44:21.500349 6903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df7a89860364bd3b23f0d8f94d727b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df7a89860364bd3b23f0d8f94d727b8" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37887 } } }
I20260810 23:44:21.500649 6904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:21.500659 6903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8 [sys.catalog]: This master's current role is: LEADER
W20260810 23:44:21.507037 6934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:44:21.507170 6934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:44:21.507320 6935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:21.509411 6935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:21.517644 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:21.497092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40385 (local address 127.30.33.62:37887)
0810 23:44:21.497299 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:21.497306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:21.499514 (+ 2208us) server_negotiation.cc:407] Connection header received
0810 23:44:21.499842 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:21.499850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:21.499937 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:21.500015 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:21.502201 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:21.503436 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:21.505168 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:21.505885 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:21.506058 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:21.509511 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:21.509519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:21.509525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:21.509567 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:21.513625 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:21.514693 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:21.514699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:21.514703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:21.514891 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:21.515504 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:21.515510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:21.515513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:21.515943 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:21.516235 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:21.517114 (+ 879us) server_negotiation.cc:299] Negotiation successful
0810 23:44:21.517324 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:44:21.519731 6935 catalog_manager.cc:1347] Generated new cluster ID: 22ca0adb0e514d4eb66fe4df2b7f4b32
I20260810 23:44:21.519840 6935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:21.531386 6935 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:21.532800 6935 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:21.538391 6935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1df7a89860364bd3b23f0d8f94d727b8: Generated new TSK 0
I20260810 23:44:21.539247 6935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:21.549373 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6744
I20260810 23:44:21.551677 6878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:21.643550 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6744
W20260810 23:44:21.664650 6718 connection.cc:535] client connection to 127.30.33.62:37887 recv error: Network error: failed to read from TLS socket (remote: 127.30.33.62:37887): Connection reset by peer (error 104)
2026-08-10T23:44:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:21 dist-test-slave-pww0 krb5kdc[6954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:21 dist-test-slave-pww0 krb5kdc[6954](info): set up 2 sockets
Aug 10 23:44:21 dist-test-slave-pww0 krb5kdc[6954](info): commencing operation
krb5kdc: starting...
W20260810 23:44:22.958271 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:23 dist-test-slave-pww0 krb5kdc[6954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:23.326241 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37263
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:23.679286 6975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:23.679636 6975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:23.679777 6975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:23.687227 6975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:23.687357 6975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:23.687419 6975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:23.687536 6975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:23.687597 6975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:23.697786 6975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:37263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:23.699174 6975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:23.700518 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:23.712132 6986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:23.712185 6992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:23.712208 6995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:23.714026 6975 server_base.cc:1034] running on GCE node
I20260810 23:44:23.714962 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:23.716552 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:23.717789 6975 hybrid_clock.cc:648] HybridClock initialized: now 1786405463717743 us; error 35 us; skew 500 ppm
I20260810 23:44:23.718147 6975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:23 dist-test-slave-pww0 krb5kdc[6954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405463, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:23.721437 6975 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:23.723038 6975 webserver.cc:460] Webserver started at http://127.30.33.62:34703/ using document root <none> and password file <none>
I20260810 23:44:23.723830 6975 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:23.723956 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:23.724263 6975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:23.727378 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "7b2614b45c82475a8010d89623afbf3c"
format_stamp: "Formatted at 2026-08-10 23:44:23 on dist-test-slave-pww0"
server_key: "0ad7426492ee590eb9e3a5d525a2bdf4"
server_key_iv: "35dbf64a906e5e537f73b621a50581c0"
server_key_version: "encryptionkey@0"
I20260810 23:44:23.728236 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "7b2614b45c82475a8010d89623afbf3c"
format_stamp: "Formatted at 2026-08-10 23:44:23 on dist-test-slave-pww0"
server_key: "0ad7426492ee590eb9e3a5d525a2bdf4"
server_key_iv: "35dbf64a906e5e537f73b621a50581c0"
server_key_version: "encryptionkey@0"
I20260810 23:44:23.733682 6975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:44:23.737213 7006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:23.738587 6975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:44:23.738770 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "7b2614b45c82475a8010d89623afbf3c"
format_stamp: "Formatted at 2026-08-10 23:44:23 on dist-test-slave-pww0"
server_key: "0ad7426492ee590eb9e3a5d525a2bdf4"
server_key_iv: "35dbf64a906e5e537f73b621a50581c0"
server_key_version: "encryptionkey@0"
I20260810 23:44:23.738957 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:23.754065 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:23.758461 6975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:23.758750 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:23.777333 6975 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:37263
I20260810 23:44:23.777369 7091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:37263 every 8 connection(s)
I20260810 23:44:23.778831 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:23.782836 7093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:23.782961 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 6975
I20260810 23:44:23.783143 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:23.783758 30852 external_mini_cluster.cc:1428] Setting key 20fd684eb8c4732493c98fff0f8897de
I20260810 23:44:23.797560 7093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c: Bootstrap starting.
I20260810 23:44:23.798036 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:23.785714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52135 (local address 127.30.33.62:37263)
0810 23:44:23.786619 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:23.786631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:23.786684 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:44:23.788331 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:23.788355 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:23.788816 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:23.789176 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:23.789896 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:23.791244 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:23.793182 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:23.793930 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:23.794072 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:23.795496 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:23.795512 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:23.795536 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:23.796139 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:23.796231 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:23.796264 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:23.796402 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:23.796713 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 23:44:23.796914 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":451,"threads_started":1}
I20260810 23:44:23.801030 7093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:23.802134 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:23.802166 7093 log.cc:824] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:23.805549 7093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c: No bootstrap required, opened a new log
I20260810 23:44:23.814368 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:23.807325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37347 (local address 127.30.33.62:37263)
0810 23:44:23.807583 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:23.807589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:23.808026 (+ 437us) server_negotiation.cc:407] Connection header received
0810 23:44:23.808190 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:23.808195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:23.808296 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:23.808424 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:23.808927 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:23.810097 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:23.811536 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:23.812248 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:23.812377 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:23.813493 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:23.813500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:23.813505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:23.813530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:23.813587 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:23.813612 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:23.813735 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:23.813934 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 23:44:23.813998 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:44:23.815620 7093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2614b45c82475a8010d89623afbf3c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37263 } }
I20260810 23:44:23.816154 7093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:23.816277 7093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2614b45c82475a8010d89623afbf3c, State: Initialized, Role: FOLLOWER
I20260810 23:44:23.816886 7093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2614b45c82475a8010d89623afbf3c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37263 } }
I20260810 23:44:23.817102 7093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:23.817204 7093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:23.817329 7093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:23.819603 7093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2614b45c82475a8010d89623afbf3c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37263 } }
I20260810 23:44:23.820108 7093 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2614b45c82475a8010d89623afbf3c; no voters:
I20260810 23:44:23.820667 7093 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:23.820848 7116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:23.821564 7116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [term 1 LEADER]: Becoming Leader. State: Replica: 7b2614b45c82475a8010d89623afbf3c, State: Running, Role: LEADER
I20260810 23:44:23.822476 7116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2614b45c82475a8010d89623afbf3c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37263 } }
I20260810 23:44:23.823100 7093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:23.825335 7118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2614b45c82475a8010d89623afbf3c. Latest consensus state: current_term: 1 leader_uuid: "7b2614b45c82475a8010d89623afbf3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2614b45c82475a8010d89623afbf3c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37263 } } }
I20260810 23:44:23.825744 7118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:23.826768 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2614b45c82475a8010d89623afbf3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2614b45c82475a8010d89623afbf3c" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 37263 } } }
I20260810 23:44:23.827108 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:23.833065 7126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:23.838613 7126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:23.848412 7126 catalog_manager.cc:1347] Generated new cluster ID: ea8534b175074dd3a5dd0c25c8e54177
I20260810 23:44:23.848541 7126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:23.863794 7126 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:23.865796 7126 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:23.874512 7126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2614b45c82475a8010d89623afbf3c: Generated new TSK 0
I20260810 23:44:23.875710 7126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:23.883425 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6975
W20260810 23:44:23.884883 7086 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52135
W20260810 23:44:23.885722 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 6975: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:23.885890 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 6975
W20260810 23:44:23.911659 6950 connection.cc:535] client connection to 127.30.33.62:37263 recv error: Network error: failed to read from TLS socket (remote: 127.30.33.62:37263): Connection reset by peer (error 104)
2026-08-10T23:44:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:24 dist-test-slave-pww0 krb5kdc[7156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:24 dist-test-slave-pww0 krb5kdc[7156](info): set up 2 sockets
Aug 10 23:44:24 dist-test-slave-pww0 krb5kdc[7156](info): commencing operation
krb5kdc: starting...
W20260810 23:44:25.233839 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:25 dist-test-slave-pww0 krb5kdc[7156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:25.594823 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:39041
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:25.949811 7182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:25.950166 7182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:25.950349 7182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:25.957791 7182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:25.957928 7182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:25.957988 7182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:25.958091 7182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:25.958148 7182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:25.968606 7182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:39041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:25.970072 7182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:25.971544 7182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:25.984229 7188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:25.984157 7192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:25.984210 7194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:25.985272 7182 server_base.cc:1034] running on GCE node
I20260810 23:44:25.986186 7182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:25.987839 7182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:25.989068 7182 hybrid_clock.cc:648] HybridClock initialized: now 1786405465989020 us; error 45 us; skew 500 ppm
I20260810 23:44:25.989483 7182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:25 dist-test-slave-pww0 krb5kdc[7156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405465, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:25.996654 7182 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:25.998458 7182 webserver.cc:460] Webserver started at http://127.30.33.62:40549/ using document root <none> and password file <none>
I20260810 23:44:25.999266 7182 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:25.999408 7182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:25.999775 7182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:26.003053 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "8d646bf919324edaa6623a27da521f7f"
format_stamp: "Formatted at 2026-08-10 23:44:25 on dist-test-slave-pww0"
server_key: "582fa29918bfc0b8fd722039f392efda"
server_key_iv: "70740025e159d2e65a710c04a2bebdc9"
server_key_version: "encryptionkey@0"
I20260810 23:44:26.003959 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "8d646bf919324edaa6623a27da521f7f"
format_stamp: "Formatted at 2026-08-10 23:44:25 on dist-test-slave-pww0"
server_key: "582fa29918bfc0b8fd722039f392efda"
server_key_iv: "70740025e159d2e65a710c04a2bebdc9"
server_key_version: "encryptionkey@0"
I20260810 23:44:26.009852 7182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:44:26.013880 7206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:26.015519 7182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:44:26.015756 7182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "8d646bf919324edaa6623a27da521f7f"
format_stamp: "Formatted at 2026-08-10 23:44:25 on dist-test-slave-pww0"
server_key: "582fa29918bfc0b8fd722039f392efda"
server_key_iv: "70740025e159d2e65a710c04a2bebdc9"
server_key_version: "encryptionkey@0"
I20260810 23:44:26.015969 7182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:26.022601 7182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:26.027238 7182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:26.027572 7182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:26.046072 7182 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:39041
I20260810 23:44:26.046147 7304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:39041 every 8 connection(s)
I20260810 23:44:26.047605 7182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:26.050136 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7182
I20260810 23:44:26.050359 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:26.050783 30852 external_mini_cluster.cc:1428] Setting key 720588b33295ea92d7580a13d9b8c5f0
I20260810 23:44:26.051905 7305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:26.059777 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:26.052710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36097 (local address 127.30.33.62:39041)
0810 23:44:26.053604 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:26.053620 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:26.053675 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:44:26.055538 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:26.055570 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:26.056085 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:26.056590 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:26.057621 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:26.057630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:26.057649 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:26.057672 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:26.057759 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:26.057790 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:26.058047 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:26.058194 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 23:44:26.058410 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":340,"threads_started":1}
I20260810 23:44:26.064011 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:26.067242 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f: Bootstrap starting.
I20260810 23:44:26.071501 7305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:26.072018 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:26.069545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40849 (local address 127.30.33.62:39041)
0810 23:44:26.069743 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:26.069750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:26.070066 (+ 316us) server_negotiation.cc:407] Connection header received
0810 23:44:26.070213 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:26.070218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:26.070300 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:26.070423 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:26.071322 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:26.071329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:26.071333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:26.071357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:26.071411 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:26.071434 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:26.071592 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:26.071658 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 23:44:26.071724 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:44:26.072887 7305 log.cc:824] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:26.076122 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f: No bootstrap required, opened a new log
I20260810 23:44:26.083827 7305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d646bf919324edaa6623a27da521f7f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39041 } }
I20260810 23:44:26.084220 7305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:26.084327 7305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d646bf919324edaa6623a27da521f7f, State: Initialized, Role: FOLLOWER
I20260810 23:44:26.084949 7305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d646bf919324edaa6623a27da521f7f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39041 } }
I20260810 23:44:26.085172 7305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:26.085307 7305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:26.085465 7305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:26.088035 7305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d646bf919324edaa6623a27da521f7f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39041 } }
I20260810 23:44:26.088610 7305 leader_election.cc:302] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d646bf919324edaa6623a27da521f7f; no voters:
I20260810 23:44:26.089228 7305 leader_election.cc:288] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:26.089389 7335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:26.090039 7335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [term 1 LEADER]: Becoming Leader. State: Replica: 8d646bf919324edaa6623a27da521f7f, State: Running, Role: LEADER
I20260810 23:44:26.090649 7335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d646bf919324edaa6623a27da521f7f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39041 } }
I20260810 23:44:26.091346 7305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:26.094120 7341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d646bf919324edaa6623a27da521f7f. Latest consensus state: current_term: 1 leader_uuid: "8d646bf919324edaa6623a27da521f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d646bf919324edaa6623a27da521f7f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39041 } } }
I20260810 23:44:26.094553 7341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:26.094448 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d646bf919324edaa6623a27da521f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d646bf919324edaa6623a27da521f7f" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 39041 } } }
I20260810 23:44:26.094802 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:26.096345 7347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:26.102211 7347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:26.112519 7347 catalog_manager.cc:1347] Generated new cluster ID: db52ba0f7038422ead34d9515c5eb0ac
I20260810 23:44:26.112661 7347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:26.160557 7347 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:26.161158 7347 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:26.169374 7347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d646bf919324edaa6623a27da521f7f: Generated new TSK 0
I20260810 23:44:26.170368 7347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:26.177211 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7182
W20260810 23:44:26.178547 7293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36097
W20260810 23:44:26.179347 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7182: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:26.179555 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7182
2026-08-10T23:44:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:26 dist-test-slave-pww0 krb5kdc[7373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:26 dist-test-slave-pww0 krb5kdc[7373](info): set up 2 sockets
Aug 10 23:44:26 dist-test-slave-pww0 krb5kdc[7373](info): commencing operation
krb5kdc: starting...
W20260810 23:44:27.504678 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.002s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:27 dist-test-slave-pww0 krb5kdc[7373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:27.871496 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40149
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:28.233196 7403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:28.233543 7403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:28.233680 7403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:28.241199 7403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:28.241349 7403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:28.241432 7403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:28.241506 7403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:28.241571 7403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:28.252076 7403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:40149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:28.253512 7403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:28.254937 7403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:28.267336 7411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:28.267372 7412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:28.267400 7419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:28.268177 7403 server_base.cc:1034] running on GCE node
I20260810 23:44:28.269014 7403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:28.270612 7403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:28.271868 7403 hybrid_clock.cc:648] HybridClock initialized: now 1786405468271811 us; error 52 us; skew 500 ppm
I20260810 23:44:28.272306 7403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:28 dist-test-slave-pww0 krb5kdc[7373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405468, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:28.275686 7403 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:28.277328 7403 webserver.cc:460] Webserver started at http://127.30.33.62:43815/ using document root <none> and password file <none>
I20260810 23:44:28.278082 7403 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:28.278211 7403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:28.278527 7403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:28.281798 7403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "a7aa847a1ec743908c99ab58be29d59e"
format_stamp: "Formatted at 2026-08-10 23:44:28 on dist-test-slave-pww0"
server_key: "4c8db1adc9c7ddc5d143d5b753848f3d"
server_key_iv: "98c5c82f4dd4aefd159145a352ca8145"
server_key_version: "encryptionkey@0"
I20260810 23:44:28.282631 7403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "a7aa847a1ec743908c99ab58be29d59e"
format_stamp: "Formatted at 2026-08-10 23:44:28 on dist-test-slave-pww0"
server_key: "4c8db1adc9c7ddc5d143d5b753848f3d"
server_key_iv: "98c5c82f4dd4aefd159145a352ca8145"
server_key_version: "encryptionkey@0"
I20260810 23:44:28.288347 7403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:44:28.292218 7437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:28.293802 7403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:44:28.294014 7403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "a7aa847a1ec743908c99ab58be29d59e"
format_stamp: "Formatted at 2026-08-10 23:44:28 on dist-test-slave-pww0"
server_key: "4c8db1adc9c7ddc5d143d5b753848f3d"
server_key_iv: "98c5c82f4dd4aefd159145a352ca8145"
server_key_version: "encryptionkey@0"
I20260810 23:44:28.294229 7403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:28.306196 7403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:28.310921 7403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:28.311196 7403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:28.329033 7403 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:40149
I20260810 23:44:28.329067 7532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:40149 every 8 connection(s)
I20260810 23:44:28.330572 7403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:28.334650 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:28.338532 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7403
I20260810 23:44:28.338737 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:28.339200 30852 external_mini_cluster.cc:1428] Setting key 66a79b87e3edf7effb69ff9d79aea517
I20260810 23:44:28.348939 7533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e: Bootstrap starting.
I20260810 23:44:28.352881 7533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:28.353983 7533 log.cc:824] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:44:28 dist-test-slave-pww0 krb5kdc[7373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:28.357488 7533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e: No bootstrap required, opened a new log
I20260810 23:44:28.365295 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:28.341375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55455 (local address 127.30.33.62:40149)
0810 23:44:28.342282 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:28.342299 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:28.342352 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:44:28.344392 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:28.344428 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:28.345088 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:28.345630 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:28.347033 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:28.349123 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:28.350523 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:28.351659 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:28.351848 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:28.356141 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:28.356154 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:28.356185 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:28.356241 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:28.360269 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:28.361084 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:28.361094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:28.361104 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:28.361354 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:28.361748 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:28.361753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:28.361756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:28.362511 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:28.362827 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:28.363208 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 23:44:28.363773 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":380,"threads_started":1}
I20260810 23:44:28.368049 7533 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aa847a1ec743908c99ab58be29d59e" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40149 } }
I20260810 23:44:28.368477 7533 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:28.368582 7533 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7aa847a1ec743908c99ab58be29d59e, State: Initialized, Role: FOLLOWER
I20260810 23:44:28.369230 7533 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aa847a1ec743908c99ab58be29d59e" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40149 } }
I20260810 23:44:28.369506 7533 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:28.369647 7533 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:28.369849 7533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:28.369985 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:28.372287 7533 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aa847a1ec743908c99ab58be29d59e" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40149 } }
I20260810 23:44:28.372975 7533 leader_election.cc:302] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7aa847a1ec743908c99ab58be29d59e; no voters:
I20260810 23:44:28.373634 7533 leader_election.cc:288] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:28.373762 7548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:28.374542 7548 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [term 1 LEADER]: Becoming Leader. State: Replica: a7aa847a1ec743908c99ab58be29d59e, State: Running, Role: LEADER
I20260810 23:44:28.375259 7548 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aa847a1ec743908c99ab58be29d59e" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40149 } }
I20260810 23:44:28.377142 7533 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:28.380139 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7aa847a1ec743908c99ab58be29d59e. Latest consensus state: current_term: 1 leader_uuid: "a7aa847a1ec743908c99ab58be29d59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aa847a1ec743908c99ab58be29d59e" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40149 } } }
I20260810 23:44:28.380550 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:28.379865 7550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7aa847a1ec743908c99ab58be29d59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aa847a1ec743908c99ab58be29d59e" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 40149 } } }
I20260810 23:44:28.381183 7550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:28.396023 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:28.374822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56891 (local address 127.30.33.62:40149)
0810 23:44:28.375029 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:28.375035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:28.376363 (+ 1328us) server_negotiation.cc:407] Connection header received
0810 23:44:28.376645 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:28.376650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:28.376737 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:28.376825 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:28.378364 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:28.379559 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:28.381372 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:28.382210 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:28.382363 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:28.386375 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:28.386384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:28.386390 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:28.386446 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:28.390719 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:28.391961 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:28.391968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:28.391973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:28.392170 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:28.392721 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:28.392726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:28.392729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:28.393245 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:28.393523 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:28.395053 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0810 23:44:28.395360 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:44:28.402498 7559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:28.409101 7559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:28.419675 7559 catalog_manager.cc:1347] Generated new cluster ID: 585e118b0c314ce19ec44919fd109929
I20260810 23:44:28.419821 7559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:28.433073 7559 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:28.435124 7559 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:28.449352 7559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7aa847a1ec743908c99ab58be29d59e: Generated new TSK 0
I20260810 23:44:28.450644 7559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:28.457921 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7403
I20260810 23:44:28.459625 7528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:28.551481 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7403
2026-08-10T23:44:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:28 dist-test-slave-pww0 krb5kdc[7596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:28 dist-test-slave-pww0 krb5kdc[7596](info): set up 2 sockets
Aug 10 23:44:28 dist-test-slave-pww0 krb5kdc[7596](info): commencing operation
krb5kdc: starting...
W20260810 23:44:29.860742 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:30 dist-test-slave-pww0 krb5kdc[7596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:30.227340 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35475
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:30.605350 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:30.605726 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:30.605885 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:30.613505 7621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:30.613682 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:30.613776 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:30.613867 7621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:30.613933 7621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:30.624396 7621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:35475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:35475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:30.625896 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:30.627378 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:30.640214 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:30.640364 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:30.640807 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:30.641141 7621 server_base.cc:1034] running on GCE node
I20260810 23:44:30.642031 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:30.643657 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:30.644887 7621 hybrid_clock.cc:648] HybridClock initialized: now 1786405470644821 us; error 55 us; skew 500 ppm
I20260810 23:44:30.645267 7621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:30 dist-test-slave-pww0 krb5kdc[7596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405470, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:30.648708 7621 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:30.650478 7621 webserver.cc:460] Webserver started at http://127.30.33.62:46021/ using document root <none> and password file <none>
I20260810 23:44:30.651232 7621 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:30.651350 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:30.651726 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:30.655169 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "b183ba5b902a493c92accb58e4429804"
format_stamp: "Formatted at 2026-08-10 23:44:30 on dist-test-slave-pww0"
server_key: "6bf0b8b25e67a2ae8a7f39be179ea960"
server_key_iv: "35e7ba2d7fd0797ac442bb1604518852"
server_key_version: "encryptionkey@0"
I20260810 23:44:30.656247 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "b183ba5b902a493c92accb58e4429804"
format_stamp: "Formatted at 2026-08-10 23:44:30 on dist-test-slave-pww0"
server_key: "6bf0b8b25e67a2ae8a7f39be179ea960"
server_key_iv: "35e7ba2d7fd0797ac442bb1604518852"
server_key_version: "encryptionkey@0"
I20260810 23:44:30.662467 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:44:30.666894 7657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:30.668758 7621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:44:30.668954 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "b183ba5b902a493c92accb58e4429804"
format_stamp: "Formatted at 2026-08-10 23:44:30 on dist-test-slave-pww0"
server_key: "6bf0b8b25e67a2ae8a7f39be179ea960"
server_key_iv: "35e7ba2d7fd0797ac442bb1604518852"
server_key_version: "encryptionkey@0"
I20260810 23:44:30.669153 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:30.686115 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:30.691223 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:30.691531 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:30.710207 7621 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:35475
I20260810 23:44:30.710258 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:35475 every 8 connection(s)
I20260810 23:44:30.711704 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:30.714679 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7621
I20260810 23:44:30.714877 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:30.715529 30852 external_mini_cluster.cc:1428] Setting key 41da9298744d8884a05513943db4834a
I20260810 23:44:30.716197 7747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:30.729823 7747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804: Bootstrap starting.
I20260810 23:44:30.730098 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:30.718270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33185 (local address 127.30.33.62:35475)
0810 23:44:30.719118 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:30.719130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:30.719178 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:44:30.721118 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:30.721145 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:30.721599 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:30.722021 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:30.722614 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:30.724011 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:30.725370 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:30.726134 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:30.726353 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:30.727641 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:30.727650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:30.727667 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:30.728273 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:30.728361 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:30.728393 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:30.728501 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:30.728753 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:44:30.729044 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":387,"threads_started":1}
I20260810 23:44:30.733305 7747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:30.733987 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:30.734515 7747 log.cc:824] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:30.737839 7747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804: No bootstrap required, opened a new log
I20260810 23:44:30.745051 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:30.738581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52723 (local address 127.30.33.62:35475)
0810 23:44:30.738752 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:30.738757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:30.739293 (+ 536us) server_negotiation.cc:407] Connection header received
0810 23:44:30.739519 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:30.739523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:30.739592 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:30.739687 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:30.740317 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:30.741096 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:30.742505 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:30.743010 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:30.743141 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:30.744272 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:30.744277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:30.744283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:30.744304 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:30.744346 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:30.744362 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:30.744444 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:30.744597 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:44:30.744728 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:44:30.746322 7747 raft_consensus.cc:348] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b183ba5b902a493c92accb58e4429804" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35475 } }
I20260810 23:44:30.746817 7747 raft_consensus.cc:374] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:30.746932 7747 raft_consensus.cc:729] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b183ba5b902a493c92accb58e4429804, State: Initialized, Role: FOLLOWER
I20260810 23:44:30.747612 7747 consensus_queue.cc:260] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b183ba5b902a493c92accb58e4429804" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35475 } }
I20260810 23:44:30.747865 7747 raft_consensus.cc:388] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:30.747998 7747 raft_consensus.cc:482] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:30.748157 7747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:30.750461 7747 raft_consensus.cc:504] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b183ba5b902a493c92accb58e4429804" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35475 } }
I20260810 23:44:30.751011 7747 leader_election.cc:302] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b183ba5b902a493c92accb58e4429804; no voters:
I20260810 23:44:30.751644 7747 leader_election.cc:288] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:30.751771 7771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:30.752964 7771 raft_consensus.cc:686] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [term 1 LEADER]: Becoming Leader. State: Replica: b183ba5b902a493c92accb58e4429804, State: Running, Role: LEADER
I20260810 23:44:30.753554 7771 consensus_queue.cc:237] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b183ba5b902a493c92accb58e4429804" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35475 } }
I20260810 23:44:30.754063 7747 sys_catalog.cc:564] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:30.756408 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b183ba5b902a493c92accb58e4429804. Latest consensus state: current_term: 1 leader_uuid: "b183ba5b902a493c92accb58e4429804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b183ba5b902a493c92accb58e4429804" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35475 } } }
I20260810 23:44:30.756405 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b183ba5b902a493c92accb58e4429804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b183ba5b902a493c92accb58e4429804" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 35475 } } }
I20260810 23:44:30.756870 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:30.756870 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:30.757771 7782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:30.763514 7782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:30.773829 7782 catalog_manager.cc:1347] Generated new cluster ID: faa3a9383c244815b5fe770897d59478
I20260810 23:44:30.774006 7782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:30.796587 7782 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:30.798175 7782 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:30.807564 7782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b183ba5b902a493c92accb58e4429804: Generated new TSK 0
I20260810 23:44:30.808525 7782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:30.815550 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7621
W20260810 23:44:30.816864 7739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33185
W20260810 23:44:30.817703 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7621: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:30.817888 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7621
2026-08-10T23:44:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:30 dist-test-slave-pww0 krb5kdc[7806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:30 dist-test-slave-pww0 krb5kdc[7806](info): set up 2 sockets
Aug 10 23:44:30 dist-test-slave-pww0 krb5kdc[7806](info): commencing operation
krb5kdc: starting...
W20260810 23:44:32.163408 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.006s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:32 dist-test-slave-pww0 krb5kdc[7806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:32.538949 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46351
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:46351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:32.898783 7830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:32.899140 7830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:32.899288 7830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:32.906787 7830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:32.906921 7830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:32.906991 7830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:32.907085 7830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:32.907145 7830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:32.917539 7830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:46351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:32.918928 7830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:32.920375 7830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:32.932369 7847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:32.932426 7851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:32.932466 7846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:32.933497 7830 server_base.cc:1034] running on GCE node
I20260810 23:44:32.934388 7830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:32.936038 7830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:32.937268 7830 hybrid_clock.cc:648] HybridClock initialized: now 1786405472937215 us; error 40 us; skew 500 ppm
I20260810 23:44:32.937683 7830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:32 dist-test-slave-pww0 krb5kdc[7806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405472, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:32.940956 7830 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:32.942564 7830 webserver.cc:460] Webserver started at http://127.30.33.62:36911/ using document root <none> and password file <none>
I20260810 23:44:32.943284 7830 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:32.943408 7830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:32.943763 7830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:32.946983 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "e3928de6158640f489daa6424085d387"
format_stamp: "Formatted at 2026-08-10 23:44:32 on dist-test-slave-pww0"
server_key: "1c823dc2c677354dfa6a8136cc5092a6"
server_key_iv: "e756f9c89fe08bd6a0320e6c5bc850c6"
server_key_version: "encryptionkey@0"
I20260810 23:44:32.947855 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "e3928de6158640f489daa6424085d387"
format_stamp: "Formatted at 2026-08-10 23:44:32 on dist-test-slave-pww0"
server_key: "1c823dc2c677354dfa6a8136cc5092a6"
server_key_iv: "e756f9c89fe08bd6a0320e6c5bc850c6"
server_key_version: "encryptionkey@0"
I20260810 23:44:32.953691 7830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:44:32.957726 7862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:32.959280 7830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:44:32.959527 7830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "e3928de6158640f489daa6424085d387"
format_stamp: "Formatted at 2026-08-10 23:44:32 on dist-test-slave-pww0"
server_key: "1c823dc2c677354dfa6a8136cc5092a6"
server_key_iv: "e756f9c89fe08bd6a0320e6c5bc850c6"
server_key_version: "encryptionkey@0"
I20260810 23:44:32.959755 7830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:32.966228 7830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:32.971192 7830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:32.971560 7830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:32.991058 7830 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:46351
I20260810 23:44:32.991099 7960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:46351 every 8 connection(s)
I20260810 23:44:32.992601 7830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:32.995651 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 7830
I20260810 23:44:32.995818 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:32.996263 30852 external_mini_cluster.cc:1428] Setting key 36a817e8ec5d1f67d040ab1ce67ab88c
I20260810 23:44:32.996786 7961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:33.006165 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:32.998197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48543 (local address 127.30.33.62:46351)
0810 23:44:32.999087 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:32.999103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:32.999158 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:44:33.001784 (+ 2626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:33.001820 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:33.002375 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:33.002926 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:33.004093 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:33.004102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:33.004123 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:33.004144 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:33.004223 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:33.004269 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:33.004516 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:33.004944 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 23:44:33.005131 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":352,"threads_started":1}
I20260810 23:44:33.010689 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:33.011420 7961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387: Bootstrap starting.
I20260810 23:44:33.014989 7961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:33.016398 7961 log.cc:824] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:33.017998 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:33.015149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43651 (local address 127.30.33.62:46351)
0810 23:44:33.015345 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:33.015351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:33.016042 (+ 691us) server_negotiation.cc:407] Connection header received
0810 23:44:33.016189 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:33.016194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:33.016264 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:33.016389 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:33.017337 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:33.017341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:33.017345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:44:33.017359 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:33.017405 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:44:33.017421 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:33.017537 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:33.017660 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 23:44:33.017725 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:44:33.019748 7961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387: No bootstrap required, opened a new log
I20260810 23:44:33.027311 7961 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3928de6158640f489daa6424085d387" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46351 } }
I20260810 23:44:33.027758 7961 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:44:33.027861 7961 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3928de6158640f489daa6424085d387, State: Initialized, Role: FOLLOWER
I20260810 23:44:33.028497 7961 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3928de6158640f489daa6424085d387" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46351 } }
I20260810 23:44:33.028708 7961 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:44:33.028851 7961 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:44:33.028995 7961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:33.031558 7961 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3928de6158640f489daa6424085d387" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46351 } }
I20260810 23:44:33.032110 7961 leader_election.cc:302] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3928de6158640f489daa6424085d387; no voters:
I20260810 23:44:33.032774 7961 leader_election.cc:288] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:44:33.032946 7977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:33.033595 7977 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [term 1 LEADER]: Becoming Leader. State: Replica: e3928de6158640f489daa6424085d387, State: Running, Role: LEADER
I20260810 23:44:33.034202 7977 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3928de6158640f489daa6424085d387" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46351 } }
I20260810 23:44:33.034885 7961 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:33.037458 7981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3928de6158640f489daa6424085d387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3928de6158640f489daa6424085d387" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46351 } } }
I20260810 23:44:33.037812 7981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:33.039286 7996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:44:33.039369 7982 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3928de6158640f489daa6424085d387. Latest consensus state: current_term: 1 leader_uuid: "e3928de6158640f489daa6424085d387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3928de6158640f489daa6424085d387" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46351 } } }
I20260810 23:44:33.039789 7982 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:33.044370 7996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:33.053611 7996 catalog_manager.cc:1347] Generated new cluster ID: 6281bd1290454be79ea53160fc14faef
I20260810 23:44:33.053745 7996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:33.072118 7996 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:33.072726 7996 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:33.080670 7996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3928de6158640f489daa6424085d387: Generated new TSK 0
I20260810 23:44:33.081844 7996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:33.088465 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7830
W20260810 23:44:33.089823 7951 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48543
W20260810 23:44:33.090718 30852 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 7830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:44:33.090891 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 7830
2026-08-10T23:44:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2268 ms)
[----------] 18 tests from AuthTokenIssuingTest (41411 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:33 dist-test-slave-pww0 krb5kdc[8019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:33 dist-test-slave-pww0 krb5kdc[8019](info): set up 2 sockets
Aug 10 23:44:33 dist-test-slave-pww0 krb5kdc[8019](info): commencing operation
krb5kdc: starting...
W20260810 23:44:34.425194 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.004s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:34 dist-test-slave-pww0 krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:34.794939 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41565
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41565,127.30.33.61:46759,127.30.33.60:41275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:35.160187 8049 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:44:35.160545 8049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:35.160620 8049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:35.160703 8049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:35.168258 8049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:35.168375 8049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:35.168437 8049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:35.168490 8049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:35.168555 8049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:35.178941 8049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41565,127.30.33.61:46759,127.30.33.60:41275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:35.180481 8049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:35.181843 8049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:35.193854 8058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:35.193904 8059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:35.194325 8062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:35.194584 8049 server_base.cc:1034] running on GCE node
I20260810 23:44:35.195407 8049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:35.196986 8049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:35.198253 8049 hybrid_clock.cc:648] HybridClock initialized: now 1786405475198174 us; error 50 us; skew 500 ppm
I20260810 23:44:35.198647 8049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:35 dist-test-slave-pww0 krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:35.202207 8049 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:35.203815 8049 webserver.cc:460] Webserver started at http://127.30.33.62:36191/ using document root <none> and password file <none>
I20260810 23:44:35.204528 8049 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:35.204619 8049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:35.204921 8049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:35.208072 8049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "3ba8f5516d9a400cafca9767fafab325"
format_stamp: "Formatted at 2026-08-10 23:44:35 on dist-test-slave-pww0"
server_key: "e76726690f5bb4455ee36dfb4bccfa91"
server_key_iv: "4f5d2064f974ef90573177dd57a8d7ac"
server_key_version: "encryptionkey@0"
I20260810 23:44:35.208864 8049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "3ba8f5516d9a400cafca9767fafab325"
format_stamp: "Formatted at 2026-08-10 23:44:35 on dist-test-slave-pww0"
server_key: "e76726690f5bb4455ee36dfb4bccfa91"
server_key_iv: "4f5d2064f974ef90573177dd57a8d7ac"
server_key_version: "encryptionkey@0"
I20260810 23:44:35.214563 8049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 23:44:35.218650 8072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:35.220402 8049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:44:35.220649 8049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "3ba8f5516d9a400cafca9767fafab325"
format_stamp: "Formatted at 2026-08-10 23:44:35 on dist-test-slave-pww0"
server_key: "e76726690f5bb4455ee36dfb4bccfa91"
server_key_iv: "4f5d2064f974ef90573177dd57a8d7ac"
server_key_version: "encryptionkey@0"
I20260810 23:44:35.220906 8049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:35.253768 8049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:35.258838 8049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:35.259152 8049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:35.279014 8049 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:41565
I20260810 23:44:35.279027 8166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:41565 every 8 connection(s)
I20260810 23:44:35.280519 8049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:35.282320 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8049
I20260810 23:44:35.282562 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:35.283407 30852 external_mini_cluster.cc:1428] Setting key cd4d0c4325719e6f74c947d161e6d0bb
I20260810 23:44:35.285410 8168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:35.294692 8168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:35 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.33.61@KRBTEST.COM; defaulting to no policy
I20260810 23:44:35.322145 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.298913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41565 (local address 127.0.0.1:49840)
0810 23:44:35.300394 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.300443 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:35.302729 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:35.303800 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:35.303808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:35.304685 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:35.305183 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.305211 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.306924 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.306930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.308585 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.308602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.309629 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.309636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.309810 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:35.310564 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:35.310595 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:35.313551 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:35.317373 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:35.317389 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:35.317403 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:35.318034 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:35.318515 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:35.318521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:35.318524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:35.318758 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:35.319625 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:35.319635 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:35.319944 (+ 309us) client_negotiation.cc:769] Sending connection context
0810 23:44:35.320275 (+ 331us) client_negotiation.cc:240] Negotiation successful
0810 23:44:35.321005 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":130,"thread_start_us":916,"threads_started":1}
I20260810 23:44:35.322129 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.298913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.30.33.62:41565)
0810 23:44:35.299910 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:35.299930 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:35.300599 (+ 669us) server_negotiation.cc:407] Connection header received
0810 23:44:35.303209 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:35.303215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:35.303534 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:35.303723 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:35.305451 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.306740 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.308767 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.309378 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.309528 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:35.313858 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:35.313868 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:35.313883 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:35.313920 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:35.317189 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:35.318203 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:35.318212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:35.318214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:35.318388 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:35.318863 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:35.318866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:35.318868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:35.319436 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:35.319656 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:35.320444 (+ 788us) server_negotiation.cc:299] Negotiation successful
0810 23:44:35.320832 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":737,"thread_start_us":434,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.61@KRBTEST.COM" created.
I20260810 23:44:35.325706 8168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:35.327050 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.326596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37682)
0810 23:44:35.326779 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.326915 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 23:44:35.327430 8075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
I20260810 23:44:35.329404 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.328902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37690)
0810 23:44:35.329116 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.329218 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 23:44:35.329674 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:46759: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab.
Entry for principal kudu/127.30.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab.
I20260810 23:44:35.366335 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:46759
--webserver_interface=127.30.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab
--principal=kudu/127.30.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41565,127.30.33.61:46759,127.30.33.60:41275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:44:35.379123 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } attempt: 1
I20260810 23:44:35.380517 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.379982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37694)
0810 23:44:35.380235 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.380357 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 23:44:35.381640 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.381189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37700)
0810 23:44:35.381406 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.381490 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:44:35.381853 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:46759: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
I20260810 23:44:35.450138 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } attempt: 2
I20260810 23:44:35.451346 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.450890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37714)
0810 23:44:35.451101 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.451198 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:44:35.452579 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.452180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37730)
0810 23:44:35.452332 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.452430 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:44:35.452816 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:46759: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
I20260810 23:44:35.544085 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } attempt: 3
I20260810 23:44:35.545425 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.544940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37746)
0810 23:44:35.545166 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.545267 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:44:35.546525 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.546158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37752)
0810 23:44:35.546285 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.546361 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:44:35.546739 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:46759: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
I20260810 23:44:35.690009 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } attempt: 4
I20260810 23:44:35.691355 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.690779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37764)
0810 23:44:35.691122 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.691220 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260810 23:44:35.692474 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.692088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37774)
0810 23:44:35.692254 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.692342 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 23:44:35.692677 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:46759: Network error: Client connection negotiation failed: client connection to 127.30.33.61:46759: connect: Connection refused (error 111)
W20260810 23:44:35.733703 8180 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:44:35.734086 8180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:35.734189 8180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:35.734313 8180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:35.741956 8180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:35.742081 8180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:35.742175 8180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:35.742252 8180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:35.742340 8180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:35.752828 8180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41565,127.30.33.61:46759,127.30.33.60:41275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab
--principal=kudu/127.30.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:46759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:35.754207 8180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:35.755617 8180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:35.767765 8188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:35.767724 8193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:35.767704 8191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:35.768779 8180 server_base.cc:1034] running on GCE node
I20260810 23:44:35.769620 8180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:35.771144 8180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:35.772418 8180 hybrid_clock.cc:648] HybridClock initialized: now 1786405475772328 us; error 46 us; skew 500 ppm
I20260810 23:44:35.772825 8180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:35 dist-test-slave-pww0 krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:35.776180 8180 init.cc:377] Logged in from keytab as kudu/127.30.33.61@KRBTEST.COM (short username kudu)
I20260810 23:44:35.777798 8180 webserver.cc:460] Webserver started at http://127.30.33.61:40573/ using document root <none> and password file <none>
I20260810 23:44:35.778543 8180 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:35.778636 8180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:35.778916 8180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:35.782112 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data/instance:
uuid: "25249101cd6f4296ad897f67a3065b42"
format_stamp: "Formatted at 2026-08-10 23:44:35 on dist-test-slave-pww0"
server_key: "3b135cad5fa47bd9066c0528df194f1d"
server_key_iv: "e81dc1a569415f6429726dd078709705"
server_key_version: "encryptionkey@0"
I20260810 23:44:35.782900 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/wal/instance:
uuid: "25249101cd6f4296ad897f67a3065b42"
format_stamp: "Formatted at 2026-08-10 23:44:35 on dist-test-slave-pww0"
server_key: "3b135cad5fa47bd9066c0528df194f1d"
server_key_iv: "e81dc1a569415f6429726dd078709705"
server_key_version: "encryptionkey@0"
I20260810 23:44:35.788549 8180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260810 23:44:35.792557 8208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:35.794046 8180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:44:35.794222 8180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/wal
uuid: "25249101cd6f4296ad897f67a3065b42"
format_stamp: "Formatted at 2026-08-10 23:44:35 on dist-test-slave-pww0"
server_key: "3b135cad5fa47bd9066c0528df194f1d"
server_key_iv: "e81dc1a569415f6429726dd078709705"
server_key_version: "encryptionkey@0"
I20260810 23:44:35.794450 8180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:35.814656 8180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:35.819268 8180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:35.819584 8180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:35.839059 8180 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.61:46759
I20260810 23:44:35.839083 8298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.61:46759 every 8 connection(s)
I20260810 23:44:35.840567 8180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
I20260810 23:44:35.844543 8299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:35.844703 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8180
I20260810 23:44:35.844880 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-1/wal/instance
I20260810 23:44:35.845309 30852 external_mini_cluster.cc:1428] Setting key 11397687758e51f32c462f02f5336537
I20260810 23:44:35.851686 8299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:35 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:35.874387 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.854592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.30.33.62:41565)
0810 23:44:35.855122 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:35.855127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:35.856353 (+ 1226us) server_negotiation.cc:407] Connection header received
0810 23:44:35.858536 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:35.858543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:35.858634 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:35.858747 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:35.860644 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.861437 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.863108 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.863621 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.863756 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:35.867752 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:35.867759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:35.867763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:35.867820 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:35.870714 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:35.871667 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:35.871673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:35.871676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:35.871822 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:35.872344 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:35.872348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:35.872350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:35.872688 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:35.872857 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:35.873699 (+ 842us) server_negotiation.cc:299] Negotiation successful
0810 23:44:35.873953 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":279,"threads_started":1}
I20260810 23:44:35.875517 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.855245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41565 (local address 127.0.0.1:49842)
0810 23:44:35.856168 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.856213 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:35.858195 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:35.859045 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:35.859054 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:35.859698 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:35.860385 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.860405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.861644 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.861648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.862929 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.862944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.863876 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.863884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.864157 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:35.864900 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:35.864927 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:35.867405 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:35.870914 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:35.870926 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:35.870942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:35.871511 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:35.871946 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:35.871953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:35.871957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:35.872204 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:35.872838 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:35.872847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:35.873270 (+ 423us) client_negotiation.cc:769] Sending connection context
0810 23:44:35.873545 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 23:44:35.874143 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":365,"threads_started":1}
WARNING: no policy specified for kudu/127.30.33.60@KRBTEST.COM; defaulting to no policy
I20260810 23:44:35.877625 8299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.60@KRBTEST.COM" created.
Aug 10 23:44:35 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:44:35.893698 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.878457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37784)
0810 23:44:35.878647 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.878668 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:35.878775 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:35.879937 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:35.879942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:35.880498 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:35.880650 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.880660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.882190 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.882195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.883083 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.883096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.884048 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.884055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.884257 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:35.884869 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:35.884888 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:35.887082 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:35.890730 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:35.890736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:35.890740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:35.891268 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:35.891795 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:35.891799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:35.891802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:35.891974 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:35.892717 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:35.892722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:35.892870 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 23:44:35.893011 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:44:35.893340 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:44:35.894057 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.878713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.30.33.61:46759)
0810 23:44:35.879367 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:35.879373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:35.879407 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:44:35.879492 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:35.879543 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:35.879782 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:35.879944 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:35.880830 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.882042 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.883227 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.883882 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.884023 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:35.887334 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:35.887348 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:35.887365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:35.887404 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:35.890546 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:35.891427 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:35.891435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:35.891438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:35.891659 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:35.892106 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:35.892111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:35.892113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:35.892557 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:35.892747 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:35.893180 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 23:44:35.893507 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":494,"thread_start_us":340,"threads_started":1}
I20260810 23:44:35.896734 8299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:35.897907 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.897439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41892)
0810 23:44:35.897618 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.897753 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 23:44:35.898151 8210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:35.899905 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.899533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41894)
0810 23:44:35.899684 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.899786 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:44:35.900110 8299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab.
Entry for principal kudu/127.30.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab.
I20260810 23:44:35.923633 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:41275
--webserver_interface=127.30.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab
--principal=kudu/127.30.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:41565,127.30.33.61:46759,127.30.33.60:41275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:44:35.949460 8299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 1
I20260810 23:44:35.950667 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.950158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41904)
0810 23:44:35.950383 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.950497 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 23:44:35.951717 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.951320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41908)
0810 23:44:35.951504 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.951599 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:44:35.952010 8299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:35.991940 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } attempt: 5
Aug 10 23:44:35 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:44:36.006204 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.992912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.30.33.61:46759)
0810 23:44:35.993101 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:35.993108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:35.993131 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:44:35.993267 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:35.993271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:35.993358 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:35.993449 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:35.994290 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.995000 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.996242 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.996692 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.996813 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.000486 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.000492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.000495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.000534 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.003432 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.004195 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.004201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.004204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.004338 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.004731 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.004735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.004738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.005034 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.005194 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.005631 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.005873 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:44:36.006201 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:35.992762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37792)
0810 23:44:35.992965 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:35.992988 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:35.993127 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:35.993503 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:35.993508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:35.993933 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:35.994120 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.994130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.995142 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.995148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.996070 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:35.996083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:35.996846 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:35.996852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:35.997077 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:35.997756 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:35.997775 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:36.000264 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:36.003621 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.003628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.003632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.004061 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.004455 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.004460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.004462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.004612 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.005156 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:36.005161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:36.005331 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 23:44:36.005497 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 23:44:36.005810 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:44:36.007272 8168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:36.008525 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.008113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41918)
0810 23:44:36.008292 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.008372 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 23:44:36.009502 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.009163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41928)
0810 23:44:36.009312 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.009389 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:44:36.009708 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:36.020295 8299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 2
I20260810 23:44:36.021632 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.021069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41940)
0810 23:44:36.021300 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.021423 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 23:44:36.022789 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.022388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41946)
0810 23:44:36.022543 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.022636 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:44:36.023036 8299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:36.060956 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 1
I20260810 23:44:36.062143 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.061662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41948)
0810 23:44:36.061873 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.061971 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:44:36.063370 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.062934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41962)
0810 23:44:36.063128 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.063221 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 23:44:36.063652 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:36.114310 8299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 3
I20260810 23:44:36.115800 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.115149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41968)
0810 23:44:36.115427 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.115596 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 23:44:36.117185 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.116629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41978)
0810 23:44:36.116912 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.117026 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260810 23:44:36.117453 8299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:36.131870 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 2
I20260810 23:44:36.133158 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.132656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41982)
0810 23:44:36.132890 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.132988 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 23:44:36.134317 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.133898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41990)
0810 23:44:36.134073 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.134171 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 23:44:36.134565 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:36.240792 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 3
I20260810 23:44:36.242208 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.241579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41996)
0810 23:44:36.241854 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.241973 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 23:44:36.243520 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.242990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:41998)
0810 23:44:36.243251 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.243341 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260810 23:44:36.243748 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:36.260707 8299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 4
I20260810 23:44:36.262212 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.261552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:42004)
0810 23:44:36.261834 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.261993 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 23:44:36.263355 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.262944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:42012)
0810 23:44:36.263101 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.263209 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:44:36.263624 8299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
W20260810 23:44:36.299989 8312 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:44:36.300349 8312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:36.300426 8312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:36.300510 8312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:36.308104 8312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:36.308228 8312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:36.308296 8312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:36.308346 8312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:36.308413 8312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:36.319032 8312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:41565,127.30.33.61:46759,127.30.33.60:41275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab
--principal=kudu/127.30.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:41275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:36.320523 8312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:36.321910 8312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:36.334892 8318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:36.334800 8325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:36.334847 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:36.335872 8312 server_base.cc:1034] running on GCE node
I20260810 23:44:36.336766 8312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:36.338411 8312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:36.339658 8312 hybrid_clock.cc:648] HybridClock initialized: now 1786405476339573 us; error 54 us; skew 500 ppm
I20260810 23:44:36.340077 8312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405476, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:36.349349 8312 init.cc:377] Logged in from keytab as kudu/127.30.33.60@KRBTEST.COM (short username kudu)
I20260810 23:44:36.351214 8312 webserver.cc:460] Webserver started at http://127.30.33.60:34615/ using document root <none> and password file <none>
I20260810 23:44:36.352077 8312 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:36.352183 8312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:36.352532 8312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:36.355870 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data/instance:
uuid: "b049690139e9485885121b47c91ae30b"
format_stamp: "Formatted at 2026-08-10 23:44:36 on dist-test-slave-pww0"
server_key: "b670d18ac214a0ea84740a045fc8e931"
server_key_iv: "93ec395ceecc867591c0c0de2bcbc47f"
server_key_version: "encryptionkey@0"
I20260810 23:44:36.356760 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/wal/instance:
uuid: "b049690139e9485885121b47c91ae30b"
format_stamp: "Formatted at 2026-08-10 23:44:36 on dist-test-slave-pww0"
server_key: "b670d18ac214a0ea84740a045fc8e931"
server_key_iv: "93ec395ceecc867591c0c0de2bcbc47f"
server_key_version: "encryptionkey@0"
I20260810 23:44:36.362447 8312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:44:36.366454 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:36.367988 8312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:44:36.368173 8312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/wal
uuid: "b049690139e9485885121b47c91ae30b"
format_stamp: "Formatted at 2026-08-10 23:44:36 on dist-test-slave-pww0"
server_key: "b670d18ac214a0ea84740a045fc8e931"
server_key_iv: "93ec395ceecc867591c0c0de2bcbc47f"
server_key_version: "encryptionkey@0"
I20260810 23:44:36.368403 8312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:36.409614 8312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:36.414258 8312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:36.414551 8312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:36.420993 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 4
I20260810 23:44:36.422676 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.421789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:42016)
0810 23:44:36.422203 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.422390 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260810 23:44:36.423848 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.423396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:42024)
0810 23:44:36.423581 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.423676 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 23:44:36.424098 8168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:41275: Network error: Client connection negotiation failed: client connection to 127.30.33.60:41275: connect: Connection refused (error 111)
I20260810 23:44:36.434094 8312 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.60:41275
I20260810 23:44:36.434118 8442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.60:41275 every 8 connection(s)
I20260810 23:44:36.435596 8312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
I20260810 23:44:36.439865 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:36.441707 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8312
I20260810 23:44:36.441900 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786405091056293-30852-0/minicluster-data/master-2/wal/instance
I20260810 23:44:36.442337 30852 external_mini_cluster.cc:1428] Setting key 9c5afba0e83e8ac0ae5e202e75e2c31b
I20260810 23:44:36.451340 8443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405476, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:36.469352 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.444525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.30.33.60:41275)
0810 23:44:36.445526 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.445550 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.445623 (+ 73us) server_negotiation.cc:407] Connection header received
0810 23:44:36.447617 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.447658 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.448316 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.448897 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.450422 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.452613 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.454055 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.455087 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.455330 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.460779 (+ 5449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.460790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.460806 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.460868 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.464344 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.465269 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.465279 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.465290 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.465480 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.465835 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.465840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.465844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.466604 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.466926 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.467400 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.467923 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":411,"threads_started":1}
I20260810 23:44:36.471673 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.454678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.30.33.62:41565)
0810 23:44:36.455332 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.455339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.455986 (+ 647us) server_negotiation.cc:407] Connection header received
0810 23:44:36.456167 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.456172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.456251 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.456361 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.457983 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.458750 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.460642 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.461286 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.461442 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.465167 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.465173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.465177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.465221 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.468632 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.469423 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.469429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.469432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.469599 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.470043 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.470047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.470049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.470326 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.470447 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.471013 (+ 566us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.471230 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":289,"threads_started":1}
I20260810 23:44:36.471695 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.454636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:41565 (local address 127.0.0.1:49854)
0810 23:44:36.455843 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.455884 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:36.456043 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:36.456449 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:36.456456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:36.457409 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:36.457779 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.457795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.458927 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.458931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.460453 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.460467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.461492 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.461497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.461691 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.462287 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:36.462314 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:36.464955 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:36.468792 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.468807 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.468813 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.469270 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.469738 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.469742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.469746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.469933 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.470462 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:36.470470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:36.470669 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 23:44:36.470862 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 23:44:36.471215 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":636,"threads_started":1}
I20260810 23:44:36.473325 8443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405476, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:36.486932 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.474079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:46759 (local address 127.0.0.1:37804)
0810 23:44:36.474236 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.474254 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:36.474336 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:36.474618 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:36.474621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:36.474967 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:36.475102 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.475109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.476163 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.476168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.477055 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.477067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.477748 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.477753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.477928 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.478510 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:36.478527 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:36.480659 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:36.484398 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.484402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.484405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.484805 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.485240 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.485245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.485248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.485437 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.486021 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:36.486026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:36.486242 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 23:44:36.486384 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 23:44:36.486617 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 23:44:36.487143 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.474115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.30.33.61:46759)
0810 23:44:36.474313 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.474319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.474340 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:44:36.474423 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.474428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.474504 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.474614 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.475252 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.476026 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.477197 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.477600 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.477718 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.480860 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.480866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.480869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.480906 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.484239 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.484928 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.484934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.484937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.485113 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.485536 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.485539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.485541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.485897 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.486047 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.486519 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.486792 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:44:36.487999 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.472854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.30.33.62:41565)
0810 23:44:36.473426 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.473432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.473448 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:44:36.473496 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.473502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.473577 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.473647 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.475110 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.475891 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.477088 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.477485 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.477596 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.481745 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.481753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.481758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.481800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.484900 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.485739 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.485744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.485747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.485896 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.486309 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.486313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.486315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.486652 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.486962 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.487434 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.487706 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456}
I20260810 23:44:36.488472 8443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405476, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:44:36.503665 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.489331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:42040)
0810 23:44:36.489498 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.489513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:36.489642 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:36.490147 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:36.490151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:36.490592 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:36.490735 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.490743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.492040 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.492050 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.493242 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.493255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.494230 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.494235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.494470 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.495120 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:36.495149 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:36.497585 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:36.501217 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.501221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.501224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.501607 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.501974 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.501979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.501983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.502110 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.502623 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:36.502628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:36.502780 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 23:44:36.502907 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:44:36.503212 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 23:44:36.503712 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.489372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.30.33.60:41275)
0810 23:44:36.489914 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.489918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.489934 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:44:36.489974 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.489978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.490031 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.490106 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.490885 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.491892 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.493395 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.494033 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.494171 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.497799 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.497804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.497807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.497839 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.501070 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.501738 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.501743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.501745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.501857 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.502219 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.502223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.502225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.502511 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.502645 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.503027 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.503387 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":431}
I20260810 23:44:36.505975 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.489758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.30.33.61:46759)
0810 23:44:36.489928 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.489933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.489950 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:44:36.490059 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.490064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.490133 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.490232 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.491702 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.492747 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.494330 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.494939 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.495077 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.499860 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.499867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.499872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.499944 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.502828 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.503589 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.503595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.503599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.503779 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.504289 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.504294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.504297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.504660 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.504999 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.505401 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.505643 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:44:36.509663 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: Bootstrap starting.
I20260810 23:44:36.511686 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:36.513752 8443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:36.515295 8443 log.cc:824] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:36.518435 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: No bootstrap required, opened a new log
I20260810 23:44:36.515547 8118 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3ba8f5516d9a400cafca9767fafab325"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49868
I20260810 23:44:36.526307 8443 raft_consensus.cc:348] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.526714 8443 raft_consensus.cc:729] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b049690139e9485885121b47c91ae30b, State: Initialized, Role: FOLLOWER
I20260810 23:44:36.527200 8443 consensus_queue.cc:260] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.528323 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:36.528775 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:36.529521 8443 sys_catalog.cc:564] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:44:36.539947 8476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:44:36.540112 8476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:44:36.536906 8312 master_runner.cc:186] Error getting master registration for 127.30.33.62:41565: OK, instance_id { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" instance_seqno: 1786405475273699 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:44:36.548424 8312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:44:36.562878 8299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 5
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
I20260810 23:44:36.580700 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.563756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:42048)
0810 23:44:36.563982 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.564004 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:36.564139 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:36.564491 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:36.564495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:36.565028 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:36.565258 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.565271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.566818 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.566823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.568309 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.568327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.569436 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.569442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.569686 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.570434 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:36.570459 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:36.573377 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:36.577664 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.577669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.577675 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.578254 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.578705 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.578710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.578714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.578902 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.579625 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:36.579630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:36.579793 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 23:44:36.579951 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 23:44:36.580290 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:44:36.580732 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.563830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.30.33.60:41275)
0810 23:44:36.564135 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.564141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.564167 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:44:36.564247 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.564254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.564339 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.564454 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.565445 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.566618 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.568488 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.569266 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.569402 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.573639 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.573646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.573652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.573708 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.577487 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.578411 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.578418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.578422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.578588 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.579029 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.579035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.579038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.579509 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.579730 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.580083 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.580346 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260810 23:44:36.587035 8299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: Bootstrap starting.
I20260810 23:44:36.591512 8299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:36.592939 8299 log.cc:824] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:36.596215 8299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: No bootstrap required, opened a new log
I20260810 23:44:36.607988 8299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.608484 8299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25249101cd6f4296ad897f67a3065b42, State: Initialized, Role: FOLLOWER
I20260810 23:44:36.609042 8299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.610116 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:36.610638 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:36.611434 8299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:44:36.621683 8501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:44:36.621856 8501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:44:36.619086 8180 master_runner.cc:186] Error getting master registration for 127.30.33.62:41565: OK, instance_id { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" instance_seqno: 1786405475273699 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:44:36.631781 8180 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:44:36.666980 8353 tablet.cc:2369] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:44:36.701351 8168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } attempt: 5
Aug 10 23:44:36 dist-test-slave-pww0 krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405475, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
I20260810 23:44:36.721381 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.702227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:41275 (local address 127.0.0.1:42060)
0810 23:44:36.702465 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:36.702488 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:36.702632 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:36.703284 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:36.703289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:36.703872 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:36.704098 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.704112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.705603 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.705610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.707154 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:36.707175 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.708359 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.708366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:36.708597 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.709329 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:36.709355 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:36.712548 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:36.717938 (+ 5390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.717945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.717949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.718568 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.719111 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:36.719117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:36.719122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:36.719375 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:36.720236 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:36.720242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:36.720424 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 23:44:36.720576 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 23:44:36.720894 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:44:36.722714 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.702696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.30.33.60:41275)
0810 23:44:36.702893 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.702899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.702923 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:44:36.702996 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.703001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.703089 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.703211 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.704307 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.705402 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.707352 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.708135 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.708285 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.712815 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.712828 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.712834 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.712966 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.717685 (+ 4719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.718753 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.718760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.718764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.718956 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.719597 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.719602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.719604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.719980 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.720133 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.722048 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.722335 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:44:36.729578 8168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325: Bootstrap starting.
I20260810 23:44:36.734117 8168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:36.735765 8168 log.cc:824] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:36.746292 8168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325: No bootstrap required, opened a new log
I20260810 23:44:36.754688 8168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.755170 8168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba8f5516d9a400cafca9767fafab325, State: Initialized, Role: FOLLOWER
I20260810 23:44:36.755769 8168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.762918 8505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:36.763381 8505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:36.765378 8168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:36.765669 8118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:44:36.765965 8118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:36.769637 8118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.772590 8255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ba8f5516d9a400cafca9767fafab325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25249101cd6f4296ad897f67a3065b42"
I20260810 23:44:36.773150 8255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:36.777225 8255 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba8f5516d9a400cafca9767fafab325 in term 1.
I20260810 23:44:36.778107 8075 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25249101cd6f4296ad897f67a3065b42, 3ba8f5516d9a400cafca9767fafab325; no voters:
I20260810 23:44:36.778694 8505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:36.780076 8390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ba8f5516d9a400cafca9767fafab325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b049690139e9485885121b47c91ae30b"
I20260810 23:44:36.780601 8390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:36.784577 8390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba8f5516d9a400cafca9767fafab325 in term 1.
I20260810 23:44:36.785465 8118 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [CANDIDATE]: Term 1 election: Requested vote from peers 25249101cd6f4296ad897f67a3065b42 (127.30.33.61:46759), b049690139e9485885121b47c91ae30b (127.30.33.60:41275)
I20260810 23:44:36.790581 8505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba8f5516d9a400cafca9767fafab325, State: Running, Role: LEADER
I20260810 23:44:36.791287 8505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } }
I20260810 23:44:36.809018 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba8f5516d9a400cafca9767fafab325. Latest consensus state: current_term: 1 leader_uuid: "3ba8f5516d9a400cafca9767fafab325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:36.809585 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:36.820003 8517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260810 23:44:36.822635 8216 tablet.cc:2369] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:44:36.863814 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.825663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.30.33.62:41565)
0810 23:44:36.825900 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.825908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.836184 (+ 10276us) server_negotiation.cc:407] Connection header received
0810 23:44:36.836536 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.836544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.836637 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.836728 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.839814 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.841097 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.843607 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.844371 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.844526 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.848788 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.848797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.848803 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.848861 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.858741 (+ 9880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.859965 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.859973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.859977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.860210 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.860778 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.860785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.860789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.861250 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.861625 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.863000 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.863313 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:44:36.872713 8517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:36.888098 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.832565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.30.33.60:41275)
0810 23:44:36.832771 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.832776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.842980 (+ 10204us) server_negotiation.cc:407] Connection header received
0810 23:44:36.845278 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.845285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.845374 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.845440 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.851621 (+ 6181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.852744 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.865196 (+ 12452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.866033 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.866164 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.879045 (+ 12881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.879053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.879059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.879111 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.882881 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.884948 (+ 2067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.884955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.884958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.885140 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.885669 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.885675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.885679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.886067 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.886320 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.887393 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.887714 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:44:36.896658 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:36.831857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.30.33.61:46759)
0810 23:44:36.832061 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:36.832067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:36.844768 (+ 12701us) server_negotiation.cc:407] Connection header received
0810 23:44:36.846759 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:36.846765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:36.846851 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:36.846916 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:36.854759 (+ 7843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.856010 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.866170 (+ 10160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:36.866894 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:36.867030 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:36.879710 (+ 12680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:36.879719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:36.879724 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:36.879773 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:36.885963 (+ 6190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.886942 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.886949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.886952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.887229 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:36.887784 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:36.887790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:36.887794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:36.888417 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:36.890362 (+ 1945us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:36.895820 (+ 5458us) server_negotiation.cc:299] Negotiation successful
0810 23:44:36.896213 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:44:36.925822 8390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [term 1 FOLLOWER]: Refusing update from remote peer 3ba8f5516d9a400cafca9767fafab325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:44:36.927176 8255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [term 1 FOLLOWER]: Refusing update from remote peer 3ba8f5516d9a400cafca9767fafab325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:44:36.928457 8505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:44:36.930401 8505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:44:36.982561 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba8f5516d9a400cafca9767fafab325. Latest consensus state: current_term: 1 leader_uuid: "3ba8f5516d9a400cafca9767fafab325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:36.982946 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:36.987437 8532 mvcc.cc:205] Tried to move back new op lower bound from 7317116833423679488 to 7317116832956039168. Current Snapshot: MvccSnapshot[applied={T|T < 7317116833423679488}]
I20260810 23:44:36.998894 8538 mvcc.cc:205] Tried to move back new op lower bound from 7317116833423679488 to 7317116832956039168. Current Snapshot: MvccSnapshot[applied={T|T < 7317116833423679488}]
I20260810 23:44:37.000089 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ba8f5516d9a400cafca9767fafab325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:37.000379 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:37.000738 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba8f5516d9a400cafca9767fafab325. Latest consensus state: current_term: 1 leader_uuid: "3ba8f5516d9a400cafca9767fafab325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:37.002761 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:37.006261 8517 catalog_manager.cc:1347] Generated new cluster ID: 5e47b07301194882b302ef0f5f709cd7
I20260810 23:44:37.006394 8517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:37.017707 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ba8f5516d9a400cafca9767fafab325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:37.018069 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:37.018708 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ba8f5516d9a400cafca9767fafab325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:37.018978 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:37.019382 8539 mvcc.cc:205] Tried to move back new op lower bound from 7317116833423679488 to 7317116832956039168. Current Snapshot: MvccSnapshot[applied={T|T < 7317116833423679488}]
I20260810 23:44:37.026362 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ba8f5516d9a400cafca9767fafab325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba8f5516d9a400cafca9767fafab325" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 41565 } } peers { permanent_uuid: "25249101cd6f4296ad897f67a3065b42" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 46759 } } peers { permanent_uuid: "b049690139e9485885121b47c91ae30b" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 41275 } } }
I20260810 23:44:37.026696 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:37.035413 8557 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:44:37.044126 8517 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:37.046108 8517 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:44:37.082705 8517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba8f5516d9a400cafca9767fafab325: Generated new TSK 0
I20260810 23:44:37.084233 8517 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:44:37.184932 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.172299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.30.33.61:46759)
0810 23:44:37.172550 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.172556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.181825 (+ 9269us) server_negotiation.cc:407] Connection header received
0810 23:44:37.182843 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.182849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.182951 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.183058 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.184664 (+ 1606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37836: BlockingRecv error: recv got EOF from 127.0.0.1:37836 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260810 23:44:37.184921 8451 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.173457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.30.33.60:41275)
0810 23:44:37.173658 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.173665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.182791 (+ 9126us) server_negotiation.cc:407] Connection header received
0810 23:44:37.183084 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.183089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.183173 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.183247 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.184664 (+ 1417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42076: BlockingRecv error: recv got EOF from 127.0.0.1:42076 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 23:44:37.184931 8452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.171766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.30.33.62:41565)
0810 23:44:37.172006 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.172013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.182426 (+ 10413us) server_negotiation.cc:407] Connection header received
0810 23:44:37.182999 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.183006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.183099 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.183182 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.184674 (+ 1492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49890: BlockingRecv error: recv got EOF from 127.0.0.1:49890 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 23:44:37.192576 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.190636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.30.33.61:46759)
0810 23:44:37.190842 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.190848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.191097 (+ 249us) server_negotiation.cc:407] Connection header received
0810 23:44:37.191319 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.191326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.191480 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.191604 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.192302 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37852: BlockingRecv error: recv got EOF from 127.0.0.1:37852 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 23:44:37.198691 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.197021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.30.33.61:46759)
0810 23:44:37.197225 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.197232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.197380 (+ 148us) server_negotiation.cc:407] Connection header received
0810 23:44:37.197550 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.197556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.197651 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.197779 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.198460 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37860: BlockingRecv error: recv got EOF from 127.0.0.1:37860 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 23:44:37.205715 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.203655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.30.33.61:46759)
0810 23:44:37.203884 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.203891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.204243 (+ 352us) server_negotiation.cc:407] Connection header received
0810 23:44:37.204459 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.204463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.204537 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.204623 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.205454 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37862: BlockingRecv error: recv got EOF from 127.0.0.1:37862 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 23:44:37.215718 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.213473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.30.33.61:46759)
0810 23:44:37.213736 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.213743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.214066 (+ 323us) server_negotiation.cc:407] Connection header received
0810 23:44:37.214313 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.214319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.214401 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.214512 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.215434 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37868: BlockingRecv error: recv got EOF from 127.0.0.1:37868 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
W20260810 23:44:37.229740 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.227653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.30.33.61:46759)
0810 23:44:37.227864 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.227870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.228149 (+ 279us) server_negotiation.cc:407] Connection header received
0810 23:44:37.228353 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.228358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.228455 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.228546 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.229455 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37874: BlockingRecv error: recv got EOF from 127.0.0.1:37874 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 23:44:37.252092 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.249455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.30.33.61:46759)
0810 23:44:37.249685 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.249691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.250357 (+ 666us) server_negotiation.cc:407] Connection header received
0810 23:44:37.250593 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.250600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.250747 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.250843 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.251844 (+ 1001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37890: BlockingRecv error: recv got EOF from 127.0.0.1:37890 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 23:44:37.290419 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.288071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.30.33.61:46759)
0810 23:44:37.288262 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.288269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.288874 (+ 605us) server_negotiation.cc:407] Connection header received
0810 23:44:37.289085 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.289090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.289192 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.289304 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.290151 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37896: BlockingRecv error: recv got EOF from 127.0.0.1:37896 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 23:44:37.360839 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.358688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.30.33.61:46759)
0810 23:44:37.358964 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.358971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.359325 (+ 354us) server_negotiation.cc:407] Connection header received
0810 23:44:37.359533 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.359540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.359644 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.359794 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.360564 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37910: BlockingRecv error: recv got EOF from 127.0.0.1:37910 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 23:44:37.494822 8306 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:37.492770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.30.33.61:46759)
0810 23:44:37.493092 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.493099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.493329 (+ 230us) server_negotiation.cc:407] Connection header received
0810 23:44:37.493553 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.493557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.493638 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.493733 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:37.494613 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37926: BlockingRecv error: recv got EOF from 127.0.0.1:37926 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
I20260810 23:44:37.542212 8476 catalog_manager.cc:1259] Loaded cluster ID: 5e47b07301194882b302ef0f5f709cd7
I20260810 23:44:37.542327 8476 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: loading cluster ID for follower catalog manager: success
I20260810 23:44:37.545068 8476 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: acquiring CA information for follower catalog manager: success
I20260810 23:44:37.547094 8476 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b049690139e9485885121b47c91ae30b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:44:37.624022 8501 catalog_manager.cc:1259] Loaded cluster ID: 5e47b07301194882b302ef0f5f709cd7
I20260810 23:44:37.624130 8501 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: loading cluster ID for follower catalog manager: success
I20260810 23:44:37.626649 8501 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: acquiring CA information for follower catalog manager: success
I20260810 23:44:37.628576 8501 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25249101cd6f4296ad897f67a3065b42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:44:37.763145 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:37.757547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.30.33.61:46759)
0810 23:44:37.757790 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.757796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.757986 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:44:37.758223 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.758230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.758392 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.758494 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:37.758954 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.759668 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.761111 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.761499 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.761587 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:37.762032 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:37.762553 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:37.762667 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:37.762775 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 23:44:37.762848 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:44:37.775695 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:37.769585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.30.33.60:41275)
0810 23:44:37.770050 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.770058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.770126 (+ 68us) server_negotiation.cc:407] Connection header received
0810 23:44:37.770423 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.770428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.770630 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.770788 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:37.771212 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.772084 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.773508 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.773922 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.774039 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:37.774494 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:37.775046 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:37.775161 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:37.775308 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 23:44:37.775385 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":246,"threads_started":1}
I20260810 23:44:37.794654 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:37.786676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.30.33.60:41275)
0810 23:44:37.786865 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.786872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.788118 (+ 1246us) server_negotiation.cc:407] Connection header received
0810 23:44:37.788445 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.788451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.788545 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.788621 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:37.789597 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.790800 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.792654 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.793291 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.793427 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:37.793920 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:37.794035 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:37.794132 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:37.794247 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 23:44:37.794336 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:44:37.794963 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:37.786755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.30.33.61:46759)
0810 23:44:37.786964 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.786970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.788697 (+ 1727us) server_negotiation.cc:407] Connection header received
0810 23:44:37.788885 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.788890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.788979 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.789074 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:37.789798 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.790523 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.792749 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.793406 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.793516 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:37.794163 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:37.794306 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:37.794387 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:37.794607 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 23:44:37.794698 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:44:37.795778 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:37.786411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.30.33.62:41565)
0810 23:44:37.788306 (+ 1895us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:37.788312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:37.788331 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:44:37.788456 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:37.788461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:37.788679 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:37.788908 (+ 229us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:37.789701 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.790800 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.792991 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:37.793460 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:37.793594 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:37.794644 (+ 1050us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:37.795146 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:37.795273 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:37.795377 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 23:44:37.795492 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1750,"thread_start_us":1608,"threads_started":1}
I20260810 23:44:37.800845 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8049
I20260810 23:44:37.802011 8162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:37.911672 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8049
I20260810 23:44:37.939929 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8180
I20260810 23:44:37.941272 8290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:38.053210 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8180
I20260810 23:44:38.080209 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8312
I20260810 23:44:38.081517 8429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:38.193297 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8312
2026-08-10T23:44:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5096 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:44:38 dist-test-slave-pww0 krb5kdc[8699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:44:38 dist-test-slave-pww0 krb5kdc[8699](info): set up 2 sockets
Aug 10 23:44:38 dist-test-slave-pww0 krb5kdc[8699](info): commencing operation
krb5kdc: starting...
W20260810 23:44:39.530612 30852 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.001s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/test-user.keytab.
Aug 10 23:44:39 dist-test-slave-pww0 krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
Entry for principal kudu/127.30.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab.
I20260810 23:44:39.901154 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46445
--webserver_interface=127.30.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:46445,127.30.33.61:34025,127.30.33.60:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:44:40.262156 8725 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:44:40.262519 8725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:40.262594 8725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:40.262677 8725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:40.270272 8725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:40.270390 8725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:40.270453 8725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:40.270507 8725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:40.270588 8725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:40.281078 8725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:46445,127.30.33.61:34025,127.30.33.60:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.62.keytab
--principal=kudu/127.30.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.62:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:40.282459 8725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:40.283869 8725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:40.296128 8731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:40.296168 8733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:40.296204 8736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:40.297058 8725 server_base.cc:1034] running on GCE node
I20260810 23:44:40.297950 8725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:40.299705 8725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:40.300977 8725 hybrid_clock.cc:648] HybridClock initialized: now 1786405480300888 us; error 52 us; skew 500 ppm
I20260810 23:44:40.301374 8725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:40 dist-test-slave-pww0 krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:40.305061 8725 init.cc:377] Logged in from keytab as kudu/127.30.33.62@KRBTEST.COM (short username kudu)
I20260810 23:44:40.306942 8725 webserver.cc:460] Webserver started at http://127.30.33.62:44345/ using document root <none> and password file <none>
I20260810 23:44:40.307739 8725 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:40.307839 8725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:40.308153 8725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:40.311340 8725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data/instance:
uuid: "1bace71600c04654b926083b70c49065"
format_stamp: "Formatted at 2026-08-10 23:44:40 on dist-test-slave-pww0"
server_key: "b8ae8cab9ffc54552d6bc4c1cfd5a149"
server_key_iv: "32150ef313f573df20cc95e0ace87e89"
server_key_version: "encryptionkey@0"
I20260810 23:44:40.312222 8725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/wal/instance:
uuid: "1bace71600c04654b926083b70c49065"
format_stamp: "Formatted at 2026-08-10 23:44:40 on dist-test-slave-pww0"
server_key: "b8ae8cab9ffc54552d6bc4c1cfd5a149"
server_key_iv: "32150ef313f573df20cc95e0ace87e89"
server_key_version: "encryptionkey@0"
I20260810 23:44:40.317751 8725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:44:40.321652 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:40.323184 8725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:44:40.323357 8725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/wal
uuid: "1bace71600c04654b926083b70c49065"
format_stamp: "Formatted at 2026-08-10 23:44:40 on dist-test-slave-pww0"
server_key: "b8ae8cab9ffc54552d6bc4c1cfd5a149"
server_key_iv: "32150ef313f573df20cc95e0ace87e89"
server_key_version: "encryptionkey@0"
I20260810 23:44:40.323609 8725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:40.346462 8725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:40.352672 8725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:40.352993 8725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:40.385428 8725 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.62:46445
I20260810 23:44:40.385452 8849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.62:46445 every 8 connection(s)
I20260810 23:44:40.387321 8725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/data/info.pb
I20260810 23:44:40.389242 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8725
I20260810 23:44:40.389451 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-0/wal/instance
I20260810 23:44:40.390017 30852 external_mini_cluster.cc:1428] Setting key 9284a681b5d67e7f0741eeebe5ff8b63
I20260810 23:44:40.392633 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:40.402037 8852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:40 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.33.61@KRBTEST.COM; defaulting to no policy
I20260810 23:44:40.430423 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.406497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46445 (local address 127.0.0.1:41340)
0810 23:44:40.407572 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.407625 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:40.410580 (+ 2955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:40.412216 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:40.412224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:40.413011 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:40.413612 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:40.413644 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.415299 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.415304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:40.417084 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:40.417103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.418174 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.418183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:40.418367 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:40.419083 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:40.419113 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:40.422175 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:40.425603 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:40.425618 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:40.425632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:40.426209 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:40.426714 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:40.426719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:40.426723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:40.426952 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:40.427814 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:40.427822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:40.428071 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 23:44:40.428324 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 23:44:40.428998 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":169,"thread_start_us":453,"threads_started":1}
I20260810 23:44:40.430441 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.408141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.30.33.62:46445)
0810 23:44:40.411306 (+ 3165us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:40.411316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:40.411363 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:44:40.411606 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:40.411616 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:40.411915 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:40.412099 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:40.413884 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.415099 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.417252 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.417872 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.418019 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:40.422451 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:40.422460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:40.422475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:40.422511 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:40.425433 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:40.426373 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:40.426383 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:40.426387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:40.426585 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:40.427060 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:40.427063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:40.427065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:40.427674 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:40.427881 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:40.428490 (+ 609us) server_negotiation.cc:299] Negotiation successful
0810 23:44:40.428860 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":2920,"thread_start_us":3927,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.61@KRBTEST.COM" created.
I20260810 23:44:40.433995 8852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:40.435242 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.434793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56530)
0810 23:44:40.434977 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.435099 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:44:40.435639 8753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
I20260810 23:44:40.437347 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.436942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56538)
0810 23:44:40.437095 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.437179 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:44:40.437584 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:34025: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab.
Entry for principal kudu/127.30.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab.
I20260810 23:44:40.474817 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:34025
--webserver_interface=127.30.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab
--principal=kudu/127.30.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:46445,127.30.33.61:34025,127.30.33.60:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:44:40.487021 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } attempt: 1
I20260810 23:44:40.488332 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.487838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56548)
0810 23:44:40.488050 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.488169 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:44:40.489455 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.489079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56550)
0810 23:44:40.489232 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.489312 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:44:40.490312 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:34025: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
I20260810 23:44:40.558633 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } attempt: 2
I20260810 23:44:40.560037 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.559407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56566)
0810 23:44:40.559729 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.559855 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 23:44:40.561179 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.560743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56574)
0810 23:44:40.560923 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.561015 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 23:44:40.561393 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:34025: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
I20260810 23:44:40.652681 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } attempt: 3
I20260810 23:44:40.654065 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.653486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56588)
0810 23:44:40.653744 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.653873 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 23:44:40.655329 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.654821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56598)
0810 23:44:40.655001 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.655102 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:44:40.655619 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:34025: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
I20260810 23:44:40.798938 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } attempt: 4
I20260810 23:44:40.800390 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.799743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56608)
0810 23:44:40.800096 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.800209 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260810 23:44:40.801532 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.801122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56622)
0810 23:44:40.801272 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.801378 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:44:40.801779 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.61:34025: Network error: Client connection negotiation failed: client connection to 127.30.33.61:34025: connect: Connection refused (error 111)
W20260810 23:44:40.834278 8861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:44:40.834635 8861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:40.834709 8861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:40.834793 8861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:40.842247 8861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:40.842352 8861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:40.842412 8861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:40.842464 8861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:40.842530 8861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:40.853093 8861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:46445,127.30.33.61:34025,127.30.33.60:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.61.keytab
--principal=kudu/127.30.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.61:34025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:40.854429 8861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:40.855767 8861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:40.868271 8876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:40.868320 8875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:40.868312 8879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:40.869387 8861 server_base.cc:1034] running on GCE node
I20260810 23:44:40.870280 8861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:40.871981 8861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:40.873274 8861 hybrid_clock.cc:648] HybridClock initialized: now 1786405480873183 us; error 56 us; skew 500 ppm
I20260810 23:44:40.873631 8861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:40 dist-test-slave-pww0 krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:40.876919 8861 init.cc:377] Logged in from keytab as kudu/127.30.33.61@KRBTEST.COM (short username kudu)
I20260810 23:44:40.878458 8861 webserver.cc:460] Webserver started at http://127.30.33.61:38971/ using document root <none> and password file <none>
I20260810 23:44:40.879212 8861 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:40.879326 8861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:40.879693 8861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:40.882910 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data/instance:
uuid: "a663d54d8caa4e42a404a866aa572d8b"
format_stamp: "Formatted at 2026-08-10 23:44:40 on dist-test-slave-pww0"
server_key: "994443311ce9acb65d66bd01fbda3b2b"
server_key_iv: "8d04f56371c83d584ff09115fcbedbca"
server_key_version: "encryptionkey@0"
I20260810 23:44:40.883821 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/wal/instance:
uuid: "a663d54d8caa4e42a404a866aa572d8b"
format_stamp: "Formatted at 2026-08-10 23:44:40 on dist-test-slave-pww0"
server_key: "994443311ce9acb65d66bd01fbda3b2b"
server_key_iv: "8d04f56371c83d584ff09115fcbedbca"
server_key_version: "encryptionkey@0"
I20260810 23:44:40.889559 8861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:44:40.893714 8893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:40.895257 8861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:44:40.895489 8861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/wal
uuid: "a663d54d8caa4e42a404a866aa572d8b"
format_stamp: "Formatted at 2026-08-10 23:44:40 on dist-test-slave-pww0"
server_key: "994443311ce9acb65d66bd01fbda3b2b"
server_key_iv: "8d04f56371c83d584ff09115fcbedbca"
server_key_version: "encryptionkey@0"
I20260810 23:44:40.895736 8861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:40.922957 8861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:40.927809 8861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:40.928130 8861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:40.947660 8861 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.61:34025
I20260810 23:44:40.947680 8988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.61:34025 every 8 connection(s)
I20260810 23:44:40.949081 8861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/data/info.pb
I20260810 23:44:40.952711 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 8861
I20260810 23:44:40.952898 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-1/wal/instance
I20260810 23:44:40.953059 8989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:40.953380 30852 external_mini_cluster.cc:1428] Setting key b36e691b36c3869c774c972bd1f01101
I20260810 23:44:40.960016 8989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:40 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:40.981477 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.962929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.30.33.62:46445)
0810 23:44:40.963497 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:40.963504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:40.964669 (+ 1165us) server_negotiation.cc:407] Connection header received
0810 23:44:40.966618 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:40.966624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:40.966710 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:40.966809 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:40.968547 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.969316 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.970892 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.971429 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.971602 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:40.975284 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:40.975289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:40.975292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:40.975331 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:40.978128 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:40.978960 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:40.978966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:40.978969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:40.979118 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:40.979597 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:40.979601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:40.979603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:40.979911 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:40.980059 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:40.980857 (+ 798us) server_negotiation.cc:299] Negotiation successful
0810 23:44:40.981099 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":291,"threads_started":1}
I20260810 23:44:40.982585 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.963625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46445 (local address 127.0.0.1:41354)
0810 23:44:40.964487 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.964529 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:40.966318 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:40.967093 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:40.967103 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:40.967724 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:40.968324 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:40.968342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.969480 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.969485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:40.970739 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:40.970753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.971711 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.971719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:40.972056 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:40.972794 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:40.972819 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:40.975039 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:40.978310 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:40.978321 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:40.978339 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:40.978825 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:40.979235 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:40.979241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:40.979245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:40.979445 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:40.980063 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:40.980072 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:40.980468 (+ 396us) client_negotiation.cc:769] Sending connection context
0810 23:44:40.980735 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 23:44:40.981298 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":380,"threads_started":1}
I20260810 23:44:40.984577 8989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.33.60@KRBTEST.COM" created.
Aug 10 23:44:40 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:44:41.000478 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.985503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56632)
0810 23:44:40.985727 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:40.985749 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:40.985876 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:40.986697 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:40.986705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:40.987078 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:40.987224 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:40.987233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.988886 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.988893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:40.990035 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:40.990053 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.991010 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.991020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:40.991194 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:40.991874 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:40.991896 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:40.994370 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:40.997736 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:40.997742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:40.997747 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:40.998138 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:40.998534 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:40.998540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:40.998543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:40.998690 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:40.999388 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:40.999393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:40.999584 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 23:44:40.999711 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:44:41.000064 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 23:44:41.000578 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:40.985436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.30.33.61:34025)
0810 23:44:40.986154 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:40.986162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:40.986213 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:44:40.986276 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:40.986285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:40.986514 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:40.986643 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:40.987430 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.988747 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.990202 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:40.990817 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:40.990923 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:40.994629 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:40.994646 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:40.994663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:40.994697 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:40.997570 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:40.998285 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:40.998293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:40.998295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:40.998423 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:40.998810 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:40.998814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:40.998816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:40.999254 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:40.999512 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:40.999881 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.000192 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":523,"thread_start_us":365,"threads_started":1}
I20260810 23:44:41.003377 8989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:41.004592 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.004086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33888)
0810 23:44:41.004287 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.004437 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 23:44:41.004841 8898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.006471 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.006107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33890)
0810 23:44:41.006251 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.006346 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:44:41.006726 8989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab.
Entry for principal kudu/127.30.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab.
I20260810 23:44:41.031615 30852 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
/tmp/dist-test-taskdkhI7B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:37347
--webserver_interface=127.30.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab
--principal=kudu/127.30.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.33.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.33.62:46445,127.30.33.61:34025,127.30.33.60:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:44:41.056077 8989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 1
I20260810 23:44:41.057410 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.056882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33904)
0810 23:44:41.057105 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.057239 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 23:44:41.058442 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.058068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33908)
0810 23:44:41.058240 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.058326 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:44:41.058667 8989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.101022 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } attempt: 5
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:44:41.116530 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.101812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56634)
0810 23:44:41.102049 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.102071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:41.102218 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:41.102640 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:41.102643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:41.103061 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:41.103263 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.103273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.104750 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.104756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.105658 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.105671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.106667 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.106671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.106836 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.107451 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:41.107501 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:41.109920 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:41.114000 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.114005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.114008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.114444 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.114866 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.114870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.114873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.115009 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.115718 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:41.115722 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:41.115886 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 23:44:41.116009 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:44:41.116269 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 23:44:41.116735 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.102036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.30.33.61:34025)
0810 23:44:41.102288 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.102295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.102321 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:44:41.102390 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.102395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.102479 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.102611 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.103427 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.104577 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.105832 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.106529 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.106664 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.110175 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.110183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.110187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.110232 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.113797 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.114582 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.114588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.114592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.114756 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.115118 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.115124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.115127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.115580 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.115739 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.116167 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.116419 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:44:41.117873 8852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:41.118980 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.118565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33918)
0810 23:44:41.118739 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.118822 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:44:41.119952 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.119593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33924)
0810 23:44:41.119760 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.119848 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:44:41.120148 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.126888 8989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 2
I20260810 23:44:41.128420 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.127655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33926)
0810 23:44:41.128049 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.128206 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260810 23:44:41.129942 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.129387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33938)
0810 23:44:41.129651 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.129776 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 23:44:41.130204 8989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.171392 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 1
I20260810 23:44:41.172600 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.172099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33940)
0810 23:44:41.172308 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.172427 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 23:44:41.173593 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.173265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33944)
0810 23:44:41.173414 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.173485 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:44:41.173847 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.221482 8989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 3
I20260810 23:44:41.222693 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.222181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33946)
0810 23:44:41.222418 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.222512 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:44:41.224089 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.223590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33956)
0810 23:44:41.223857 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.223952 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260810 23:44:41.224325 8989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.242084 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 2
I20260810 23:44:41.243362 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.242847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33968)
0810 23:44:41.243085 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.243192 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 23:44:41.244395 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.244064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33980)
0810 23:44:41.244201 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.244275 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 23:44:41.244627 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.350860 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 3
I20260810 23:44:41.352154 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.351674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33984)
0810 23:44:41.351878 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.351976 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 23:44:41.353428 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.352964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:33992)
0810 23:44:41.353169 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.353278 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 23:44:41.353681 8852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
I20260810 23:44:41.367599 8989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 4
I20260810 23:44:41.368912 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.368347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:34004)
0810 23:44:41.368635 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.368739 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 23:44:41.370048 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.369720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:34008)
0810 23:44:41.369849 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.369926 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:44:41.370308 8989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.33.60:37347: Network error: Client connection negotiation failed: client connection to 127.30.33.60:37347: connect: Connection refused (error 111)
W20260810 23:44:41.403363 9001 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:44:41.403855 9001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:44:41.403970 9001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:44:41.404067 9001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:44:41.411620 9001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:44:41.411724 9001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:44:41.411782 9001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:44:41.411834 9001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:44:41.411897 9001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:44:41.422305 9001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.33.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.33.62:46445,127.30.33.61:34025,127.30.33.60:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/krb5kdc/kudu_127.30.33.60.keytab
--principal=kudu/127.30.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.33.60:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:44:41.423715 9001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:44:41.425078 9001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:44:41.439950 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:41.440024 9016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:44:41.439986 9014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:44:41.440842 9001 server_base.cc:1034] running on GCE node
I20260810 23:44:41.441649 9001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:44:41.443186 9001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:44:41.444409 9001 hybrid_clock.cc:648] HybridClock initialized: now 1786405481444332 us; error 54 us; skew 500 ppm
I20260810 23:44:41.444756 9001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405481, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:44:41.451808 9001 init.cc:377] Logged in from keytab as kudu/127.30.33.60@KRBTEST.COM (short username kudu)
I20260810 23:44:41.453558 9001 webserver.cc:460] Webserver started at http://127.30.33.60:36407/ using document root <none> and password file <none>
I20260810 23:44:41.454324 9001 fs_manager.cc:359] Metadata directory not provided
I20260810 23:44:41.454419 9001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:44:41.454720 9001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:44:41.457885 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data/instance:
uuid: "3bbb18c2796f490296289413ea04a217"
format_stamp: "Formatted at 2026-08-10 23:44:41 on dist-test-slave-pww0"
server_key: "26cad12ca49e022b00814da6e2434ae5"
server_key_iv: "d3e6c0dce96a313807998f8587b086d8"
server_key_version: "encryptionkey@0"
I20260810 23:44:41.458700 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/wal/instance:
uuid: "3bbb18c2796f490296289413ea04a217"
format_stamp: "Formatted at 2026-08-10 23:44:41 on dist-test-slave-pww0"
server_key: "26cad12ca49e022b00814da6e2434ae5"
server_key_iv: "d3e6c0dce96a313807998f8587b086d8"
server_key_version: "encryptionkey@0"
I20260810 23:44:41.464313 9001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:44:41.468076 9027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:44:41.469555 9001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:44:41.469729 9001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data,/tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/wal
uuid: "3bbb18c2796f490296289413ea04a217"
format_stamp: "Formatted at 2026-08-10 23:44:41 on dist-test-slave-pww0"
server_key: "26cad12ca49e022b00814da6e2434ae5"
server_key_iv: "d3e6c0dce96a313807998f8587b086d8"
server_key_version: "encryptionkey@0"
I20260810 23:44:41.469934 9001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:44:41.494099 9001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:44:41.498530 9001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:44:41.498787 9001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:44:41.517078 9001 rpc_server.cc:307] RPC server started. Bound to: 127.30.33.60:37347
I20260810 23:44:41.517087 9125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.33.60:37347 every 8 connection(s)
I20260810 23:44:41.518484 9001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/data/info.pb
I20260810 23:44:41.519676 30852 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu as pid 9001
I20260810 23:44:41.519881 30852 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdkhI7B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786405091056293-30852-0/minicluster-data/master-2/wal/instance
I20260810 23:44:41.520421 30852 external_mini_cluster.cc:1428] Setting key 0ce0fb068eb428012aab678cc86960cf
I20260810 23:44:41.523175 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:44:41.530907 8852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 4
I20260810 23:44:41.531665 9126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.62@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
I20260810 23:44:41.547451 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.522309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.30.33.60:37347)
0810 23:44:41.523089 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.523102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.523144 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:44:41.524881 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.524909 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.525320 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.525673 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.527579 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.529105 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.530464 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.531247 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.531428 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.536930 (+ 5502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.536942 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.536964 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.537027 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.541689 (+ 4662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.542960 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.542969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.542979 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.543246 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.543765 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.543770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.543773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.544673 (+ 900us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.545007 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.545532 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.546017 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":365,"threads_started":1}
I20260810 23:44:41.548668 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.532040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:34034)
0810 23:44:41.532268 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.532292 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:41.532408 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:41.532764 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:41.532769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:41.533217 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:41.533377 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.533386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.534546 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.534550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.535828 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.535847 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.536924 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.536930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.537129 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.537865 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:41.537891 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:41.540978 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:41.545788 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.545793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.545797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.546338 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.546859 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.546864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.546868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.547046 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.547735 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:41.547741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:41.547909 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 23:44:41.548045 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:44:41.548377 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:44:41.549847 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.531703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.30.33.60:37347)
0810 23:44:41.532348 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.532352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.532372 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:44:41.532520 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.532524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.532582 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.532683 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.533551 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.534395 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.536018 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.536738 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.537480 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.541305 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.541313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.541318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.541370 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.545623 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.546489 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.546493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.546496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.546730 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.547187 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.547192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.547195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.547577 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.549068 (+ 1491us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.549202 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.549430 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":478,"threads_started":1}
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405481, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:41.556037 8852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065: Bootstrap starting.
I20260810 23:44:41.560061 8852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:41.560252 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.535353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.62:46445 (local address 127.0.0.1:41358)
0810 23:44:41.542760 (+ 7407us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.542829 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:41.542995 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:41.543968 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:41.543976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:41.544706 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:41.545126 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.545139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.546475 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.546480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.548170 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.548186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.549224 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.549231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.549417 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.550040 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:41.550067 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:41.552818 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:41.556975 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.556990 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.556997 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.557574 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.558060 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.558065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.558069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.558259 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.558972 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:41.558980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:41.559189 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 23:44:41.559406 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 23:44:41.559812 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7091,"thread_start_us":6652,"threads_started":1}
I20260810 23:44:41.560684 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.535372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.30.33.62:46445)
0810 23:44:41.543538 (+ 8166us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.543542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.543558 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:44:41.543600 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.543603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.543659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.543764 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.545315 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.546286 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.548363 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.549035 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.549164 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.553227 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.553238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.553278 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.553329 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.556816 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.557749 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.557755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.557758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.557933 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.558406 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.558412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.558414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.558815 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.558984 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.559580 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.559825 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8019,"thread_start_us":7875,"threads_started":1}
I20260810 23:44:41.561441 8852 log.cc:824] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.62@KRBTEST.COM
I20260810 23:44:41.562541 9126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:41.565122 8852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065: No bootstrap required, opened a new log
I20260810 23:44:41.569746 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.551339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.30.33.62:46445)
0810 23:44:41.552254 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.552258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.552277 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:44:41.552316 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.552319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.552376 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.552468 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.554076 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.555383 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.557148 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.557601 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.558330 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.562812 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.562817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.562821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.562857 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.566489 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.567341 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.567346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.567350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.567538 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.567962 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.567967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.567970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.568402 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.568666 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.569051 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.569304 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"spinlock_wait_cycles":1920,"thread_start_us":584,"threads_started":1}
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405481, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
I20260810 23:44:41.575651 8852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
I20260810 23:44:41.576031 8852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bace71600c04654b926083b70c49065, State: Initialized, Role: FOLLOWER
I20260810 23:44:41.576601 8852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
I20260810 23:44:41.578493 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.563349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.61:34025 (local address 127.0.0.1:56644)
0810 23:44:41.563685 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.563704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:41.563808 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:41.564117 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:41.564122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:41.564514 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:41.564660 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.564669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.565903 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.565909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.566921 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.566937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.567948 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.567955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.568150 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.569603 (+ 1453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:41.569627 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:41.572171 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:41.575818 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.575824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.575827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.576367 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.576787 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.576791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.576794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.576959 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.577543 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:41.577548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:41.577716 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 23:44:41.577838 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 23:44:41.578137 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260810 23:44:41.578591 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.563564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.30.33.61:34025)
0810 23:44:41.563747 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.563753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.563773 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:44:41.563925 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.563930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.563993 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.564076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.565006 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.565752 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.567096 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.567756 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.567886 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.572412 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.572419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.572423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.572459 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.575603 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.576526 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.576531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.576536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.576672 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.577100 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.577106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.577110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.577424 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.577564 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.577950 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.578194 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:44:41.579957 9126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:44:41.583770 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.584340 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:41.585148 8852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.33.61@KRBTEST.COM
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405481, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.60@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
I20260810 23:44:41.594699 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.580592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:34046)
0810 23:44:41.580745 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.580763 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:41.580858 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:41.581273 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:41.581276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:41.581616 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:41.581797 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.581806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.582750 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.582755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.583992 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.584009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.585088 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.585094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.585303 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.585857 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:41.585879 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:41.588423 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:41.591481 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.591486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.591491 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.592008 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.592404 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.592407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.592409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.592538 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.593563 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:41.593568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:41.593753 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 23:44:41.593918 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 23:44:41.594273 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 23:44:41.594918 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.580806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.30.33.60:37347)
0810 23:44:41.580977 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.580983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.581002 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:44:41.581053 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.581058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.581147 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.581252 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.581927 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.582628 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.584152 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.584768 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.584867 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.588600 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.588605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.588608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.588636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.591320 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.592145 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.592151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.592155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.592300 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.592664 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.592670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.592674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.593407 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.593596 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.594054 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.594488 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":48}
I20260810 23:44:41.596541 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.572165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.30.33.61:34025)
0810 23:44:41.575958 (+ 3793us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.575963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.575988 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:44:41.576039 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.576043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.576119 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.576241 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.578070 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.579490 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.581473 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.582184 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.583005 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.588238 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.588247 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.588253 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.588303 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.592405 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.593190 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.593196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.593200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.593341 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.593853 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.593858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.593860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.594253 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.595733 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.595864 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.596128 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3658,"thread_start_us":3430,"threads_started":1}
I20260810 23:44:41.600737 9126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: Bootstrap starting.
I20260810 23:44:41.604529 30852 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:44:41.607820 8811 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1bace71600c04654b926083b70c49065"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41374
I20260810 23:44:41.608315 8811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:44:41.608538 8811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:44:41.610085 8758 tablet.cc:2369] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:44:41.612561 9165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:44:41.612407 8811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
W20260810 23:44:41.612694 9165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:44:41.604143 8725 master_runner.cc:186] Error getting master registration for 127.30.33.61:34025: OK, instance_id { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" instance_seqno: 1786405480942121 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:44:41.613552 8725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:44:41.613895 8811 leader_election.cc:288] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [CANDIDATE]: Term 1 election: Requested vote from peers a663d54d8caa4e42a404a866aa572d8b (127.30.33.61:34025), 3bbb18c2796f490296289413ea04a217 (127.30.33.60:37347)
I20260810 23:44:41.616335 9126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:41.617753 9126 log.cc:824] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:41.627573 9126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: No bootstrap required, opened a new log
I20260810 23:44:41.638960 9126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
I20260810 23:44:41.639447 9126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bbb18c2796f490296289413ea04a217, State: Initialized, Role: FOLLOWER
I20260810 23:44:41.640031 9126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
I20260810 23:44:41.641410 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.642457 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:41.642648 9126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:41.642729 9085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bace71600c04654b926083b70c49065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3bbb18c2796f490296289413ea04a217"
I20260810 23:44:41.643260 9085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:41.646687 9085 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bace71600c04654b926083b70c49065 in term 1.
I20260810 23:44:41.647545 8754 leader_election.cc:302] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bace71600c04654b926083b70c49065, 3bbb18c2796f490296289413ea04a217; no voters:
I20260810 23:44:41.648089 9140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:44:41.669531 8989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } attempt: 5
I20260810 23:44:41.672555 9140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [term 1 LEADER]: Becoming Leader. State: Replica: 1bace71600c04654b926083b70c49065, State: Running, Role: LEADER
I20260810 23:44:41.673194 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.649477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.30.33.62:46445)
0810 23:44:41.650485 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.650492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.650517 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:44:41.650595 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.650600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.650687 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.650805 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.653154 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.654308 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.656335 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.657063 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.657179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.661332 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.661340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.661346 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.661401 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.665419 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.666540 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.666546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.666550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.666734 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.667286 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.667291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.667295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.667733 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.667990 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.672542 (+ 4552us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.672818 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836}
I20260810 23:44:41.673617 9140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
I20260810 23:44:41.677731 9187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bace71600c04654b926083b70c49065. Latest consensus state: current_term: 1 leader_uuid: "1bace71600c04654b926083b70c49065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.679175 9187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:41.680193 9201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 23:44:41 dist-test-slave-pww0 krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405480, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.33.61@KRBTEST.COM for kudu/127.30.33.60@KRBTEST.COM
I20260810 23:44:41.682651 9201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:44:41.689153 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.651163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.30.33.61:34025)
0810 23:44:41.651348 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.651354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.651373 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:44:41.651510 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.651516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.651601 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.651723 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.656910 (+ 5187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.658044 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.662694 (+ 4650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.663343 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.663492 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.670331 (+ 6839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.670337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.670341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.670389 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.679225 (+ 8836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.685473 (+ 6248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.685480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.685483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.685646 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.686558 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.686564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.686567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.686964 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.687212 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.688460 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.688757 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:44:41.691818 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.670214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.33.60:37347 (local address 127.0.0.1:34064)
0810 23:44:41.671302 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0810 23:44:41.671322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:44:41.671427 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:44:41.671959 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:44:41.671964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:44:41.672455 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:44:41.672659 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.672671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.675210 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.675216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.676477 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:44:41.676497 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.679370 (+ 2873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.679375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:44:41.679624 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.680232 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:44:41.680256 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:44:41.682939 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:44:41.687525 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.687531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.687535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.688088 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.688846 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:44:41.688852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:44:41.688856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:44:41.689052 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:44:41.690129 (+ 1077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:44:41.690135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:44:41.690424 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 23:44:41.690644 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 23:44:41.691108 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956}
I20260810 23:44:41.693643 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.670590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.30.33.60:37347)
0810 23:44:41.670799 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.670805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.671541 (+ 736us) server_negotiation.cc:407] Connection header received
0810 23:44:41.671600 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.671605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.671696 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.671776 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.672871 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.673839 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.676686 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.677310 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.677433 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.683175 (+ 5742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.683182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.683187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.683230 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.687310 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.688296 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.688303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.688306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.688601 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.689245 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.689251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.689254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.689915 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.692337 (+ 2422us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.692581 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.693045 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:44:41.699061 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:41.651924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.30.33.60:37347)
0810 23:44:41.652101 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.652107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.663521 (+ 11414us) server_negotiation.cc:407] Connection header received
0810 23:44:41.664403 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.664410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.664517 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.664599 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.670552 (+ 5953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.671602 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.673736 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:41.674399 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:41.674522 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:41.680409 (+ 5887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:44:41.680416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:44:41.680421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:44:41.680464 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:44:41.684080 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.695752 (+ 11672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.695760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.695763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.695940 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:44:41.696768 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:44:41.696774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:44:41.696777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:44:41.697143 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:44:41.697442 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:41.698417 (+ 975us) server_negotiation.cc:299] Negotiation successful
0810 23:44:41.698672 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:44:41.700989 8989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b: Bootstrap starting.
I20260810 23:44:41.706208 8989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b: Neither blocks nor log segments found. Creating new log.
I20260810 23:44:41.708997 8989 log.cc:824] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:44:41.716343 9085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [term 1 FOLLOWER]: Refusing update from remote peer 1bace71600c04654b926083b70c49065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:44:41.717623 9140 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:44:41.728507 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bace71600c04654b926083b70c49065. Latest consensus state: current_term: 1 leader_uuid: "1bace71600c04654b926083b70c49065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.728994 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:41.729818 8989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b: No bootstrap required, opened a new log
I20260810 23:44:41.735407 9187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bace71600c04654b926083b70c49065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.735747 9187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:41.740150 9201 catalog_manager.cc:1347] Generated new cluster ID: 1d10559f395a4a1a9905029daf47f72e
I20260810 23:44:41.740290 9201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:44:41.740837 9218 mvcc.cc:205] Tried to move back new op lower bound from 7317116853096046592 to 7317116852941500416. Current Snapshot: MvccSnapshot[applied={T|T < 7317116853096046592}]
I20260810 23:44:41.741348 9001 master_runner.cc:186] Error getting master registration for 127.30.33.61:34025: OK, instance_id { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" instance_seqno: 1786405480942121 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:44:41.751595 9001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:44:41.758613 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bace71600c04654b926083b70c49065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.758921 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:41.763476 8989 raft_consensus.cc:348] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
I20260810 23:44:41.764087 8989 raft_consensus.cc:729] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a663d54d8caa4e42a404a866aa572d8b, State: Initialized, Role: FOLLOWER
I20260810 23:44:41.764753 8989 consensus_queue.cc:260] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } }
I20260810 23:44:41.765048 9201 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:44:41.766173 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.766952 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:41.767577 8989 sys_catalog.cc:564] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:44:41.767632 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bace71600c04654b926083b70c49065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a663d54d8caa4e42a404a866aa572d8b"
I20260810 23:44:41.768121 8949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:44:41.772251 8949 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bace71600c04654b926083b70c49065 in term 1.
I20260810 23:44:41.773281 8948 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [term 1 FOLLOWER]: Refusing update from remote peer 1bace71600c04654b926083b70c49065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:44:41.773464 9201 catalog_manager.cc:1504] Loading token signing keys...
W20260810 23:44:41.774390 9221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:44:41.774509 9221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:44:41.775381 9187 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:44:41.785606 9201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065: Generated new TSK 0
I20260810 23:44:41.786887 9201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:44:41.810952 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bace71600c04654b926083b70c49065. Latest consensus state: current_term: 1 leader_uuid: "1bace71600c04654b926083b70c49065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.810971 9236 sys_catalog.cc:455] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bace71600c04654b926083b70c49065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.811385 9236 sys_catalog.cc:458] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:41.811383 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:44:41.812990 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bace71600c04654b926083b70c49065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bace71600c04654b926083b70c49065" member_type: VOTER last_known_addr { host: "127.30.33.62" port: 46445 } } peers { permanent_uuid: "a663d54d8caa4e42a404a866aa572d8b" member_type: VOTER last_known_addr { host: "127.30.33.61" port: 34025 } } peers { permanent_uuid: "3bbb18c2796f490296289413ea04a217" member_type: VOTER last_known_addr { host: "127.30.33.60" port: 37347 } } }
I20260810 23:44:41.813424 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bace71600c04654b926083b70c49065 [sys.catalog]: This master's current role is: LEADER
I20260810 23:44:41.863046 9260 catalog_manager.cc:1259] Loaded cluster ID: 1d10559f395a4a1a9905029daf47f72e
I20260810 23:44:41.863266 9260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b: loading cluster ID for follower catalog manager: success
I20260810 23:44:41.867897 9260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b: acquiring CA information for follower catalog manager: success
I20260810 23:44:41.880605 9260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a663d54d8caa4e42a404a866aa572d8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:44:41.968158 9133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:41.941986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.30.33.62:46445)
0810 23:44:41.942235 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.942243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.964988 (+ 22745us) server_negotiation.cc:407] Connection header received
0810 23:44:41.965674 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.965682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.965787 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.965869 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.967851 (+ 1982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47346: BlockingRecv error: recv got EOF from 127.0.0.1:47346 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260810 23:44:41.971186 9138 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:41.944234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.30.33.61:34025)
0810 23:44:41.944475 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.944482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.966346 (+ 21864us) server_negotiation.cc:407] Connection header received
0810 23:44:41.968145 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.968152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.968261 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.968356 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.970872 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51598: BlockingRecv error: recv got EOF from 127.0.0.1:51598 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 23:44:41.975786 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:41.967133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.30.33.60:37347)
0810 23:44:41.967355 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:41.967362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:41.969102 (+ 1740us) server_negotiation.cc:407] Connection header received
0810 23:44:41.969441 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:41.969452 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:41.969567 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:41.969669 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:41.975502 (+ 5833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35866: BlockingRecv error: recv got EOF from 127.0.0.1:35866 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:44:42.029843 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.012428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.30.33.61:34025)
0810 23:44:42.012667 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.012673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.021141 (+ 8468us) server_negotiation.cc:407] Connection header received
0810 23:44:42.021810 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.021817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.022073 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.022221 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.022988 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.024165 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.026224 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.027064 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.027176 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.028102 (+ 926us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.028917 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.029056 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.029347 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.029458 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 23:44:42.078351 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.062660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.30.33.60:37347)
0810 23:44:42.062918 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.062926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.076171 (+ 13245us) server_negotiation.cc:407] Connection header received
0810 23:44:42.076483 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.076490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.076648 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.076764 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.078096 (+ 1332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35870: BlockingRecv error: recv got EOF from 127.0.0.1:35870 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 23:44:42.139734 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.120883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.30.33.61:34025)
0810 23:44:42.121106 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.121112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.132086 (+ 10974us) server_negotiation.cc:407] Connection header received
0810 23:44:42.132743 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.132750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.132855 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.132930 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.133778 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.134938 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.137069 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.137811 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.137930 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.138686 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.138881 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.138960 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.139260 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.139372 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 23:44:42.185194 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.175506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.30.33.60:37347)
0810 23:44:42.175737 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.175745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.182908 (+ 7163us) server_negotiation.cc:407] Connection header received
0810 23:44:42.183545 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.183552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.183679 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.183764 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.184934 (+ 1170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35878: BlockingRecv error: recv got EOF from 127.0.0.1:35878 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 23:44:42.236166 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.221927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.30.33.61:34025)
0810 23:44:42.222196 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.222203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.228279 (+ 6076us) server_negotiation.cc:407] Connection header received
0810 23:44:42.228637 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.228645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.228757 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.228834 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.229586 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.230782 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.233661 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.234323 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.234445 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.235201 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.235369 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.235444 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.235733 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.235840 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260810 23:44:42.283015 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.272492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.30.33.60:37347)
0810 23:44:42.272711 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.272717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.280663 (+ 7946us) server_negotiation.cc:407] Connection header received
0810 23:44:42.281297 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.281304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.281413 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.281492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.282711 (+ 1219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35880: BlockingRecv error: recv got EOF from 127.0.0.1:35880 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:44:42.306903 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.299068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.30.33.61:34025)
0810 23:44:42.299355 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.299361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.300004 (+ 643us) server_negotiation.cc:407] Connection header received
0810 23:44:42.300272 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.300279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.300385 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.300511 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.301127 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.302331 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.304501 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.305291 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.305438 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.305981 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.306192 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.306300 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.306494 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.306600 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
W20260810 23:44:42.326895 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.324279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.30.33.60:37347)
0810 23:44:42.324615 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.324621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.325221 (+ 600us) server_negotiation.cc:407] Connection header received
0810 23:44:42.325435 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.325442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.325545 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.325670 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.326646 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35894: BlockingRecv error: recv got EOF from 127.0.0.1:35894 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
I20260810 23:44:42.346549 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.338873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.30.33.61:34025)
0810 23:44:42.339107 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.339113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.339813 (+ 700us) server_negotiation.cc:407] Connection header received
0810 23:44:42.340072 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.340079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.340180 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.340309 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.340883 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.342069 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.344131 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.344821 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.344978 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.345549 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.345714 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.345804 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.345996 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.346105 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 23:44:42.356913 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.354517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.30.33.60:37347)
0810 23:44:42.354749 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.354757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.355170 (+ 413us) server_negotiation.cc:407] Connection header received
0810 23:44:42.355416 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.355426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.355555 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.355668 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.356644 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35910: BlockingRecv error: recv got EOF from 127.0.0.1:35910 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:44:42.384035 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.377548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.30.33.61:34025)
0810 23:44:42.377784 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.377790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.378130 (+ 340us) server_negotiation.cc:407] Connection header received
0810 23:44:42.378369 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.378377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.378453 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.378543 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.379002 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.379869 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.381893 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.382608 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.382747 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.383269 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.383411 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.383545 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.383662 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.383753 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 23:44:42.395817 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.393283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.30.33.60:37347)
0810 23:44:42.393480 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.393486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.394078 (+ 592us) server_negotiation.cc:407] Connection header received
0810 23:44:42.394373 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.394379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.394490 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.394612 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.395596 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35912: BlockingRecv error: recv got EOF from 127.0.0.1:35912 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:44:42.436941 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.431367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.30.33.61:34025)
0810 23:44:42.432238 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.432244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.432267 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:44:42.432343 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.432348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.432435 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.432530 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.432998 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.433804 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.435193 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.435687 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.435807 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.436219 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.436349 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.436428 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.436580 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.436666 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730}
W20260810 23:44:42.444651 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.442738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.30.33.60:37347)
0810 23:44:42.442982 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.442989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.443290 (+ 301us) server_negotiation.cc:407] Connection header received
0810 23:44:42.443418 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.443422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.443559 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.443670 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.444391 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35918: BlockingRecv error: recv got EOF from 127.0.0.1:35918 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:44:42.517928 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.512328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.30.33.61:34025)
0810 23:44:42.512633 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.512639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.512843 (+ 204us) server_negotiation.cc:407] Connection header received
0810 23:44:42.513004 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.513008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.513087 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.513171 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.513657 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.514433 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.515882 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.516326 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.516441 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.517035 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.517196 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.517273 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.517517 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.517615 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260810 23:44:42.526345 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.524242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.30.33.60:37347)
0810 23:44:42.524449 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.524456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.524889 (+ 433us) server_negotiation.cc:407] Connection header received
0810 23:44:42.525089 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.525093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.525170 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.525262 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.526089 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35926: BlockingRecv error: recv got EOF from 127.0.0.1:35926 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:44:42.663102 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.658208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.30.33.61:34025)
0810 23:44:42.658411 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.658416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.658704 (+ 288us) server_negotiation.cc:407] Connection header received
0810 23:44:42.658898 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.658902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.658976 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.659078 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.659567 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.660303 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.661636 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.662077 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.662178 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.662526 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.662638 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.662701 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.662797 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.662871 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 23:44:42.670897 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:44:42.669016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.30.33.60:37347)
0810 23:44:42.669197 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.669201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.669533 (+ 332us) server_negotiation.cc:407] Connection header received
0810 23:44:42.669713 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.669718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.669799 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.669893 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:44:42.670682 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35928: BlockingRecv error: recv got EOF from 127.0.0.1:35928 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:44:42.776434 9221 catalog_manager.cc:1259] Loaded cluster ID: 1d10559f395a4a1a9905029daf47f72e
I20260810 23:44:42.776546 9221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: loading cluster ID for follower catalog manager: success
I20260810 23:44:42.779172 9221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: acquiring CA information for follower catalog manager: success
I20260810 23:44:42.781244 9221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bbb18c2796f490296289413ea04a217: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:44:42.938309 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.933173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.30.33.61:34025)
0810 23:44:42.933413 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.933420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.933671 (+ 251us) server_negotiation.cc:407] Connection header received
0810 23:44:42.933905 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.933909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.934006 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.934090 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.934491 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.935219 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.936654 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.937084 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.937191 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.937631 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.937754 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.937823 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.937924 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.937999 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:44:42.950857 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.944430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.30.33.60:37347)
0810 23:44:42.944674 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.944680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.944919 (+ 239us) server_negotiation.cc:407] Connection header received
0810 23:44:42.945118 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.945122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.945296 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.945401 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.945847 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.946599 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.948539 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.948960 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.949070 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.949622 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.950265 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.950386 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.950493 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.950583 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:44:42.964702 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.957577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.30.33.61:34025)
0810 23:44:42.957764 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.957771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.958769 (+ 998us) server_negotiation.cc:407] Connection header received
0810 23:44:42.958935 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.958941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.959044 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.959140 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.959980 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.960901 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.962663 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.963195 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.963334 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.963905 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.964048 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.964131 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.964314 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.964404 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:44:42.965405 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.957704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.30.33.62:46445)
0810 23:44:42.958188 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.958194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.958509 (+ 315us) server_negotiation.cc:407] Connection header received
0810 23:44:42.958651 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.958658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.958939 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.959131 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.960058 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.960935 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.962941 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.963559 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.963672 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.964271 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.964770 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.964868 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.965068 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.965149 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260810 23:44:42.966622 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:44:42.958412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.30.33.60:37347)
0810 23:44:42.958595 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:44:42.958599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:44:42.959446 (+ 847us) server_negotiation.cc:407] Connection header received
0810 23:44:42.959684 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:44:42.959690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:44:42.959775 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:44:42.959874 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:44:42.960443 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.961470 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.963383 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:44:42.963982 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:44:42.964056 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:44:42.964727 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:44:42.964858 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:44:42.964918 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:44:42.966242 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0810 23:44:42.966336 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:44:42.970258 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8725
I20260810 23:44:42.971386 8844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:43.084443 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8725
I20260810 23:44:43.116801 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 8861
I20260810 23:44:43.118060 8979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:43.229446 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 8861
I20260810 23:44:43.256120 30852 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu pid 9001
I20260810 23:44:43.257362 9115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:44:43.366290 30852 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdkhI7B/build/asan/bin/kudu with pid 9001
2026-08-10T23:44:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5181 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10277 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (392281 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST